Surgical Simulation Navigation System

ABSTRACT

An interactive and dynamic surgical simulation system may be used in the context of a computer-implemented interactive surgical system. The surgical simulation system may enable enhanced navigation, such as to provide simulation support during a live surgical procedure. A processor may be configured to identify a procedure plan for a live surgical procedure and identify a stored simulation of a surgical procedure that corresponds to the procedure plan. The processor may determine, from information received during the live surgical procedure, a present portion of the live surgical procedure. And the processor may retrieve, from the stored simulation, a portion of the stored simulation that corresponds to the present portion of the live surgical procedure. The portion of the simulation that corresponds to the present portion of the live surgical procedure may then be presented for user interaction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Provisional U.S. PatentApplication No. 63/191,681, May 21, 2021, the disclosure of which isincorporated herein by reference in its entirety.

This application is related to the following, filed contemporaneously,the contents of each of which are incorporated by reference herein:

-   -   U.S. patent application Ser. No. 17/332,594, filed May 27, 2021,        (Attorney Docket No. END9338USNP1), titled METHODS FOR SURGICAL        SIMULATION    -   U.S. patent application Ser. No. 17/332,524, filed May 27, 2021,        (Attorney Docket No. END9338USNP2), titled SURGICAL SIMULATION        OBJECT RECTIFICATION SYSTEM    -   U.S. patent application Ser. No. 17/332,441, filed May 27, 2021,        (Attorney Docket No. END9338USNP4), titled SURGICAL SIMULATION        SYSTEM WITH COORDINATED IMAGINING    -   U.S. patent application Ser. No. 17/332,462, filed May 27, 2021,        (Attorney Docket No. END9338USNP5), titled SURGICAL SIMULATION        SYSTEM WITH SIMULATED SURGICAL EQUIPMENT COORDINATION    -   U.S. patent application Ser. No. 17/332,197, filed May 27, 2021,        (Attorney Docket No. END9338USNP6), titled SIMULATION-BASED        SURGICAL PROCEDURE PLANNING SYSTEM    -   U.S. patent application Ser. No. 17/332,407, filed May 27, 2021,        (Attorney Docket No. END9338USNP7), titled SIMULATION-BASED        DIRECTED SURGICAL DEVELOPMENT SYSTEM    -   U.S. patent application Ser. No. 17/332,449, filed May 27, 2021,        (Attorney Docket No. END9338USNP8), titled SURGICAL ADVERSE        EVENT SIMULATION SYSTEM    -   U.S. patent application Ser. No. 17/332,496, filed May 27, 2021,        (Attorney Docket No. END9338USNP9), titled SIMULATION-BASED        SURGICAL ANALYSIS SYSTEM    -   U.S. patent application Ser. No. 17/332,480, filed May 27, 2021,        (Attorney Docket No. END9338USNP10), titled DYNAMIC ADAPTATION        SYSTEM FOR SURGICAL SIMULATION

BACKGROUND

Surgical simulations, such as computer-based, three-dimensionalsimulations of a surgical environment and/or surgical procedure forexample, present an opportunity to advance the surgical arts. Surgicalsimulations have potential to benefit surgical training, planning,development, and the like. For example, surgical simulations may be usedto train surgeons in new procedures and/or to improve the performance ofprocedures they already know. Surgical simulations may be used as avirtual “dress rehearsal” to help a surgeon prepare for an upcomingprocedure. And surgical simulations may be used to experiment withunproven procedures and techniques.

However, surgical simulation platforms are complex systems that facemany limitations in capabilities, scope, and applicability. For example,many platforms are technology “silos,” specifically programmed andtailored to address a particular learning objective or to simulate theoperation of a singular piece of equipment, such as simulating theoperation of a surgical robot. Limitations, such as these, may dimmish aplatform's effectiveness as a tool to advance the surgical arts. Andsuch limitations may represent significant technological roadblocks tothe integration of simulation-based applications into other aspects ofthe surgical process, such a pre-operative planning, intra-operativesupport, post-operative analysis, and the like.

Accordingly, innovation in surgical simulation technology, such astechnical advancements that address surgical simulation capabilities,scope, and applicability for example, may accelerate further progress inthe surgical arts.

SUMMARY

An interactive and dynamic surgical simulation system is disclosed. Thesurgical simulation system may be used in the context of acomputer-implemented interactive surgical system. The surgicalsimulation system may enable enhanced navigation. For example, a devicemay be used to provide simulation support during a live surgicalprocedure.

The device may include a processor. The processor may be configured toidentify a procedure plan for a live surgical procedure and identify astored simulation of a surgical procedure that corresponds to theprocedure plan. The processor may determine, from information receivedduring the live surgical procedure, a present portion of the livesurgical procedure. And the processor may retrieve, from the storedsimulation, a portion of the stored simulation that corresponds to thepresent portion of the live surgical procedure. The portion of thesimulation that corresponds to the present portion of the live surgicalprocedure may then be presented for user interaction.

The procedure plan may include a set of tasks. And the stored simulationmay include information indicative of a simulated activity that isindexed according to the set of tasks. The processor may determine thepresent portion of the live surgical procedure by determining a presenttask from the procedure plan. The processor may retrieve a portion ofthe simulation that corresponds to the present portion of the livesurgical procedure by retrieving a selected portion of the informationindicative of simulated activity that is indexed the present task.

Accordingly, a visualization of the present portion of the live surgicalprocedure may be presented concurrently with a presentation of theportion of the simulation that corresponds to the present portion of thelive surgical procedure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer-implemented interactive surgicalsyste.

FIG. 2 shows an example surgical system being used to perform a surgicalprocedure in an operating room.

FIG. 3 shows an example surgical hub paired with a visualization system,a robotic system, and an intelligent instrument, in accordance with atleast one aspect of the present disclosur.

FIG. 4 illustrates a surgical data network having a communication hubconfigured to connect modular devices located in one or more operatingtheaters of a healthcare facility, or any room in a healthcare facilityspecially equipped for surgical operations, to the cloud, in accordancewith at least one aspect of the present disclosure.

FIG. 5 illustrates an example computer-implemented interactive surgicalsystem.

FIG. 6 illustrates an example surgical hub comprising a plurality ofmodules coupled to the modular control tower.

FIG. 7 is a block diagram of an example surgical simulator system.

FIG. 8 is a block diagram of an example surgical simulator system.

FIG. 9 is a block diagram depicting an example surgical simulator userinterface device.

FIG. 10 is a flow chart of an example surgical simulator operation.

FIGS. 11A-B illustrate example surgical procedural plan data structuresfor use with a computer-implemented interactive surgical system and/or asurgical simulator.

FIG. 12 illustrates a system for providing simulation support in a livesurgical procedur.

FIG. 13 illustrates time-based surgical and simulation data.

FIG. 14 illustrates a task-based indexing of the time-based surgical andsimulation data.

FIG. 15 is a flow diagram of example process for providing simulationsupport in a live surgical procedure.

DETAILED DESCRIPTION

Surgical simulation systems, devices, and methods may include aspects ofintegration with other medical equipment, data sources, processes, andinstitutions. Surgical simulation systems, devices, and methods mayinclude aspects of integration with a computer-implemented interactivesurgical system and/or with one or more elements of acomputer-implemented interactive surgical system, for example.

Referring to FIG. 1, a computer-implemented interactive surgical system100 may include one or more surgical systems 102 and a cloud-basedsystem (e.g., the cloud 104 that may include a remote server 113 coupledto a storage device 105). Each surgical system 102 may include at leastone surgical hub 106 in communication with the cloud 104 that mayinclude a remote server 113.

One or more simulation devices 103, 111 may be in communication withand/or integrated as part of the computer-implemented interactivesurgical system 100. For example, the simulation device 103 may be anelement of the one or more surgical systems 102. For example, thesimulation device 103 may be in communication with one or more surgicalhubs 106. For example, the simulation device 111 may be in communicationwith the computer-implemented interactive surgical system 100 via thecloud 104.

In one example, as illustrated in FIG. 1, the surgical system 102includes a visualization system 108, a robotic system 110, and ahandheld intelligent surgical instrument 112, which are configured tocommunicate with one another and/or the hub 106. In some aspects, asurgical system 102 may include an M number of hubs 106, an N number ofvisualization systems 108, an O number of robotic systems 110, and a Pnumber of handheld intelligent surgical instruments 112, where M, N, O,and P may be integers greater than or equal to one.

In various aspects, the visualization system 108 may include one or moreimaging sensors, one or more image-processing units, one or more storagearrays, and one or more displays that are strategically arranged withrespect to the sterile field, as illustrated in FIG. 2. In one aspect,the visualization system 108 may include an interface for HL7, PACS, andEMR. Various components of the visualization system 108 are describedunder the heading “Advanced Imaging Acquisition Module” in U.S. PatentApplication Publication No. US 2019-0200844 A1 (U.S. patent applicationSer. No. 16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING,STORAGE AND DISPLAY, filed Dec. 4, 2018, the disclosure of which isherein incorporated by reference in its entirety.

As illustrated in FIG. 2, a primary display 119 is positioned in thesterile field to be visible to an operator at the operating table 114.In addition, a visualization tower 111 is positioned outside the sterilefield. The visualization tower 111 may include a first non-steriledisplay 107 and a second non-sterile display 109, which face away fromeach other. The visualization system 108, guided by the hub 106, isconfigured to utilize the displays 107, 109, and 119 to coordinateinformation flow to operators inside and outside the sterile field. Forexample, the hub 106 may cause the visualization system 108 to display asnapshot of a surgical site, as recorded by an imaging device 124, on anon-sterile display 107 or 109, while maintaining a live feed of thesurgical site on the primary display 119. The snapshot on thenon-sterile display 107 or 109 can permit a non-sterile operator toperform a diagnostic step relevant to the surgical procedure, forexample.

In one aspect, the hub 106 may also be configured to route a diagnosticinput or feedback entered by a non-sterile operator at the visualizationtower 111 to the primary display 119 within the sterile field, where itcan be viewed by a sterile operator at the operating table. In oneexample, the input can be in the form of a modification to the snapshotdisplayed on the non-sterile display 107 or 109, which can be routed tothe primary display 119 by the hub 106.

Referring to FIG. 2, a surgical instrument 112 is being used in thesurgical procedure as part of the surgical system 102. The hub 106 mayalso be configured to coordinate information flow to a display of thesurgical instrument 112. For example, in U.S. Patent ApplicationPublication No. US 2019-0200844 A1 (U.S. patent application Ser. No.16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY, filed Dec. 4, 2018, the disclosure of which is hereinincorporated by reference in its entirety. A diagnostic input orfeedback entered by a non-sterile operator at the visualization tower111 can be routed by the hub 106 to the surgical instrument display 115within the sterile field, where it can be viewed by the operator of thesurgical instrument 112. Example surgical instruments that are suitablefor use with the surgical system 102 are described under the heading“Surgical Instrument Hardware” and in U.S. Patent ApplicationPublication No. US 2019-0200844 A1 (U.S. patent application Ser. No.16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY, filed Dec. 4, 2018, the disclosure of which is hereinincorporated by reference in its entirety, for example.

FIG. 2 depicts an example of a surgical system 102 being used to performa surgical procedure on a patient who is lying down on an operatingtable 114 in a surgical operating room 116. A robotic system 110 may beused in the surgical procedure as a part of the surgical system 102. Therobotic system 110 may include a surgeon's console 118, a patient sidecart 120 (surgical robot), and a surgical robotic hub 122. The patientside cart 120 can manipulate at least one removably coupled surgicaltool 117 through a minimally invasive incision in the body of thepatient while the surgeon views the surgical site through the surgeon'sconsole 118. An image of the surgical site can be obtained by a medicalimaging device 124, which can be manipulated by the patient side cart120 to orient the imaging device 124. The robotic hub 122 can be used toprocess the images of the surgical site for subsequent display to thesurgeon through the surgeon's console 118.

Other types of robotic systems can be readily adapted for use with thesurgical system 102. Various examples of robotic systems and surgicaltools that are suitable for use with the present disclosure aredescribed in U.S. Patent Application Publication No. US 2019-0201137 A1(U.S. patent application Ser. No. 16/209,407), titled METHOD OF ROBOTICHUB COMMUNICATION, DETECTION, AND CONTROL, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.

Various examples of cloud-based analytics that are performed by thecloud 104, and are suitable for use with the present disclosure, aredescribed in U.S. Patent Application Publication No. US 2019-0206569 A1(U.S. patent application Ser. No. 16/209,403), titled METHOD OF CLOUDBASED DATA ANALYTICS FOR USE WITH THE HUB, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.

In various aspects, the imaging device 124 may include at least oneimage sensor and one or more optical components. Suitable image sensorsmay include, but are not limited to, Charge-Coupled Device (CCD) sensorsand Complementary Metal-Oxide Semiconductor (CMOS) sensors.

The optical components of the imaging device 124 may include one or moreillumination sources and/or one or more lenses. The one or moreillumination sources may be directed to illuminate portions of thesurgical field. The one or more image sensors may receive lightreflected or refracted from the surgical field, including lightreflected or refracted from tissue and/or surgical instruments.

The one or more illumination sources may be configured to radiateelectromagnetic energy in the visible spectrum as well as the invisiblespectrum. The visible spectrum, sometimes referred to as the opticalspectrum or luminous spectrum, is that portion of the electromagneticspectrum that is visible to (i.e., can be detected by) the human eye andmay be referred to as visible light or simply light. A typical human eyewill respond to wavelengths in air that are from about 380 nm to about750 nm.

The invisible spectrum (e.g., the non-luminous spectrum) is that portionof the electromagnetic spectrum that lies below and above the visiblespectrum (i.e., wavelengths below about 380 nm and above about 750 nm).The invisible spectrum is not detectable by the human eye. Wavelengthsgreater than about 750 nm are longer than the red visible spectrum, andthey become invisible infrared (IR), microwave, and radioelectromagnetic radiation. Wavelengths less than about 380 nm areshorter than the violet spectrum, and they become invisible ultraviolet,x-ray, and gamma ray electromagnetic radiation.

In various aspects, the imaging device 124 is configured for use in aminimally invasive procedure. Examples of imaging devices suitable foruse with the present disclosure include, but not limited to, anarthroscope, angioscope, bronchoscope, choledochoscope, colonoscope,cytoscope, duodenoscope, enteroscope, esophagogastro-duodenoscope(gastroscope), endoscope, laryngoscope, nasopharyngo-neproscope,sigmoidoscope, thoracoscope, and ureteroscope.

The imaging device may employ multi-spectrum monitoring to discriminatetopography and underlying structures. A multi-spectral image is one thatcaptures image data within specific wavelength ranges across theelectromagnetic spectrum. The wavelengths may be separated by filters orby the use of instruments that are sensitive to particular wavelengths,including light from frequencies beyond the visible light range, e.g.,IR and ultraviolet. Spectral imaging can allow extraction of additionalinformation the human eye fails to capture with its receptors for red,green, and blue. The use of multi-spectral imaging is described ingreater detail under the heading “Advanced Imaging Acquisition Module”in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S.patent application Ser. No. 16/209,385), titled METHOD OF HUBCOMMUNICATION, PROCESSING, STORAGE AND DISPLAY, filed Dec. 4, 2018, thedisclosure of which is herein incorporated by reference in its entirety.Multi-spectrum monitoring can be a useful tool in relocating a surgicalfield after a surgical task is completed to perform one or more of thepreviously described tests on the treated tissue. It is axiomatic thatstrict sterilization of the operating room and surgical equipment isrequired during any surgery. The strict hygiene and sterilizationconditions required in a “surgical theater,” i.e., an operating ortreatment room, necessitate the highest possible sterility of allmedical devices and equipment. Part of that sterilization process is theneed to sterilize anything that comes in contact with the patient orpenetrates the sterile field, including the imaging device 124 and itsattachments and components. It will be appreciated that the sterilefield may be considered a specified area, such as within a tray or on asterile towel, that is considered free of microorganisms, or the sterilefield may be considered an area, immediately around a patient, who hasbeen prepared for a surgical procedure. The sterile field may includethe scrubbed team members, who are properly attired, and all furnitureand fixtures in the area.

Referring now to FIG. 3, a hub 106 is depicted in communication with avisualization system 108, a robotic system 110, and a handheldintelligent surgical instrument 112. The hub 106 includes a hub display135, an imaging module 138, a generator module 140, a communicationmodule 130, a processor module 132, a storage array 134, and anoperating-room mapping module 133. In certain aspects, as illustrated inFIG. 3, the hub 106 further includes a smoke evacuation module 126and/or a suction/irrigation module 128. During a surgical procedure,energy application to tissue, for sealing and/or cutting, is generallyassociated with smoke evacuation, suction of excess fluid, and/orirrigation of the tissue. Fluid, power, and/or data lines from differentsources are often entangled during the surgical procedure. Valuable timecan be lost addressing this issue during a surgical procedure.Detangling the lines may necessitate disconnecting the lines from theirrespective modules, which may require resetting the modules. The hubmodular enclosure 136 offers a unified environment for managing thepower, data, and fluid lines, which reduces the frequency ofentanglement between such lines. Aspects of the present disclosurepresent a surgical hub for use in a surgical procedure that involvesenergy application to tissue at a surgical site. The surgical hubincludes a hub enclosure and a combo generator module slidablyreceivable in a docking station of the hub enclosure. The dockingstation includes data and power contacts. The combo generator moduleincludes two or more of an ultrasonic energy generator component, abipolar RF energy generator component, and a monopolar RF energygenerator component that are housed in a single unit. In one aspect, thecombo generator module also includes a smoke evacuation component, atleast one energy delivery cable for connecting the combo generatormodule to a surgical instrument, at least one smoke evacuation componentconfigured to evacuate smoke, fluid, and/or particulates generated bythe application of therapeutic energy to the tissue, and a fluid lineextending from the remote surgical site to the smoke evacuationcomponent. In one aspect, the fluid line is a first fluid line and asecond fluid line extends from the remote surgical site to a suction andirrigation module slidably received in the hub enclosure. In one aspect,the hub enclosure comprises a fluid interface. Certain surgicalprocedures may require the application of more than one energy type tothe tissue. One energy type may be more beneficial for cutting thetissue, while another different energy type may be more beneficial forsealing the tissue. For example, a bipolar generator can be used to sealthe tissue while an ultrasonic generator can be used to cut the sealedtissue. Aspects of the present disclosure present a solution where a hubmodular enclosure 136 is configured to accommodate different generators,and facilitate an interactive communication therebetween. One of theadvantages of the hub modular enclosure 136 is enabling the quickremoval and/or replacement of various modules. Aspects of the presentdisclosure present a modular surgical enclosure for use in a surgicalprocedure that involves energy application to tissue. The modularsurgical enclosure includes a first energy-generator module, configuredto generate a first energy for application to the tissue, and a firstdocking station comprising a first docking port that includes first dataand power contacts, wherein the first energy-generator module isslidably movable into an electrical engagement with the power and datacontacts and wherein the first energy-generator module is slidablymovable out of the electrical engagement with the first power and datacontacts. Further to the above, the modular surgical enclosure alsoincludes a second energy-generator module con-figured to generate asecond energy, different than the first energy, for application to thetissue, and a second docking station comprising a second docking portthat includes second data and power contacts, wherein the secondenergy-generator module is slidably movable into an electricalengagement with the power and data contacts, and wherein the secondenergy-generator module is slidably movable out of the electricalengagement with the second power and data contacts. In addition, themodular surgical enclosure also includes a communication bus between thefirst docking port and the second docking port, configured to facilitatecommunication between the first energy-generator module and the secondenergy-generator module. Referring to FIG. 3, aspects of the presentdisclosure are presented for a hub modular enclosure 136 that allows themodular integration of a generator module 140, a smoke evacuation module126, and a suction/irrigation module 128. The hub modular enclosure 136further facilitates interactive communication between the modules 140,126, 128. The generator module 140 can be a generator module withintegrated monopolar, bipolar, and ultrasonic components supported in asingle housing unit slidably insertable into the hub modular enclosure136. The generator module 140 can be configured to connect to amonopolar device 142, a bipolar device 144, and an ultrasonic device146. Alternatively, the generator module 140 may comprise a series ofmonopolar, bipolar, and/or ultrasonic generator modules that interactthrough the hub modular enclosure 136. The hub modular enclosure 136 canbe configured to facilitate the insertion of multiple generators andinteractive communication between the generators docked into the hubmodular enclosure 136 so that the generators would act as a singlegenerator.

FIG. 4 illustrates a surgical data network 201 comprising a modularcommunication hub 203 configured to connect modular devices located inone or more operating theaters of a healthcare facility, or any room ina healthcare facility specially equipped for surgical operations, to acloud-based system (e.g., the cloud 204 that may include a remote server213 coupled to a storage device 205). In one aspect, the modularcommunication hub 203 comprises a network hub 207 and/or a networkswitch 209 in communication with a network router. The modularcommunication hub 203 also can be coupled to a local computer system 210to provide local computer processing and data manipulation. The surgicaldata network 201 may be configured as passive, intelligent, orswitching. A passive surgical data network serves as a conduit for thedata, enabling it to go from one device (or segment) to another and tothe cloud computing resources. An intelligent surgical data networkincludes additional features to enable the traffic passing through thesurgical data network to be monitored and to configure each port in thenetwork hub 207 or network switch 209. An intelligent surgical datanetwork may be referred to as a manageable hub or switch. A switchinghub reads the destination address of each packet and then forwards thepacket to the correct port.

Modular devices 1 a-1 n located in the operating theater may be coupledto the modular communication hub 203. The network hub 207 and/or thenetwork switch 209 may be coupled to a network router 211 to connect thedevices 1 a-1 n to the cloud 204 or the local computer system 210. Dataassociated with the devices 1 a-1 n may be transferred to cloud-basedcomputers via the router for remote data processing and manipulation.Data associated with the devices 1 a-1 n may also be transferred to thelocal computer system 210 for local data processing and manipulation.Modular devices 2 a-2 m located in the same operating theater also maybe coupled to a network switch 209. The network switch 209 may becoupled to the network hub 207 and/or the network router 211 to connectto the devices 2 a-2 m to the cloud 204. Data associated with thedevices 2 a-2 n may be transferred to the cloud 204 via the networkrouter 211 for data processing and manipulation. Data associated withthe devices 2 a-2 m may also be transferred to the local computer system210 for local data processing and manipulation.

It will be appreciated that the surgical data network 201 may beexpanded by interconnecting multiple network hubs 207 and/or multiplenetwork switches 209 with multiple network routers 211. The modularcommunication hub 203 may be contained in a modular control towerconfigured to receive multiple devices 1 a-1 n/2 a-2 m. The localcomputer system 210 also may be contained in a modular control tower.The modular communication hub 203 is connected to a display 212 todisplay images obtained by some of the devices 1 a-1 n/2 a-2 m, forexample during surgical procedures. In various aspects, the devices 1a-1 n/2 a-2 m may include, for example, various modules such as animaging module 138 coupled to an endoscope, a generator module 140coupled to an energy-based surgical device, a smoke evacuation module126, a suction/irrigation module 128, a communication module 130, aprocessor module 132, a storage array 134, a surgical device coupled toa display, and/or a non-contact sensor module, among other modulardevices that may be connected to the modular communication hub 203 ofthe surgical data network 201.

In one aspect, the surgical data network 201 may comprise a combinationof network hub(s), network switch(es), and network router(s) connectingthe devices 1 a-1 n/2 a-2 m to the cloud. Any one of or all of thedevices 1 a-1 n/2 a-2 m coupled to the network hub or network switch maycollect data in real time and transfer the data to cloud computers fordata processing and manipulation. It will be appreciated that cloudcomputing relies on sharing computing resources rather than having localservers or personal devices to handle software applications. The word“cloud” may be used as a metaphor for “the Internet,” although the termis not limited as such. Accordingly, the term “cloud computing” may beused herein to refer to “a type of Internet-based computing,” wheredifferent services-such as servers, storage, and applications—aredelivered to the modular communication hub 203 and/or computer system210 located in the surgical theater (e.g., a fixed, mobile, temporary,or field operating room or space) and to devices connected to themodular communication hub 203 and/or computer system 210 through theInternet. The cloud infrastructure may be maintained by a cloud serviceprovider. In this context, the cloud service provider may be the entitythat coordinates the usage and control of the devices 1 a-1 n/2 a-2 mlocated in one or more operating theaters. The cloud computing servicescan perform a large number of calculations based on the data gathered bysmart surgical instruments, robots, and other computerized deviceslocated in the operating theater. The hub hardware enables multipledevices or connections to be connected to a computer that communicateswith the cloud computing resources and storage.

Applying cloud computer data processing techniques on the data collectedby the devices 1 a-1 n/2 a-2 m, the surgical data network can provideimproved surgical outcomes, reduced costs, and improved patientsatisfaction. At least some of the devices 1 a-1 n/2 a-2 m may beemployed to view tissue states to assess leaks or perfusion of sealedtissue after a tissue sealing and cutting procedure. At least some ofthe devices 1 a-1 n/2 a-2 m may be employed to identify pathology, suchas the effects of diseases, using the cloud-based computing to examinedata including images of samples of body tissue for diagnostic purposes.This may include localization and margin confirmation of tissue andphenotypes. At least some of the devices 1 a-1 n/2 a-2 m may be employedto identify anatomical structures of the body using a variety of sensorsintegrated with imaging devices and techniques such as overlaying imagescaptured by multiple imaging devices. The data gathered by the devices 1a-1 n/2 a-2 m, including image data, may be transferred to the cloud 204or the local computer system 210 or both for data processing andmanipulation including image processing and manipulation. The data maybe analyzed to improve surgical procedure outcomes by determining iffurther treatment, such as the application of endoscopic intervention,emerging technologies, a targeted radiation, targeted intervention, andprecise robotics to tissue-specific sites and conditions, may bepursued. Such data analysis may further employ outcome analyticsprocessing, and using standardized approaches may provide beneficialfeedback to either confirm surgical treatments and the behavior of thesurgeon or suggest modifications to surgical treatments and the behaviorof the surgeon.

The operating theater devices 1 a-1 n may be connected to the modularcommunication hub 203 over a wired channel or a wireless channeldepending on the configuration of the devices 1 a-1 n to a network hub.The network hub 207 may be implemented, in one aspect, as a localnetwork broadcast device that works on the physical layer of the OpenSystem Interconnection (OSI) model. The network hub may provideconnectivity to the devices 1 a-1 n located in the same operatingtheater network. The network hub 207 may collect data in the form ofpackets and sends them to the router in half duplex mode. The networkhub 207 may not store any media access control/Internet Protocol(MAC/IP) to transfer the device data. Only one of the devices 1 a-1 ncan send data at a time through the network hub 207. The network hub 207may not have routing tables or intelligence regarding where to sendinformation and broadcasts all network data across each connection andto a remote server 213 (FIG. 4) over the cloud 204. The network hub 207can detect basic network errors such as collisions, but having allinformation broadcast to multiple ports can be a security risk and causebottlenecks.

The operating theater devices 2 a-2 m may be connected to a networkswitch 209 over a wired channel or a wireless channel. The networkswitch 209 works in the data link layer of the OSI model. The networkswitch 209 may be a multicast device for connecting the devices 2 a-2 mlocated in the same operating theater to the network. The network switch209 may send data in the form of frames to the network router 211 andworks in full duplex mode. Multiple devices 2 a-2 m can send data at thesame time through the network switch 209. The network switch 209 storesand uses MAC addresses of the devices 2 a-2 m to transfer data.

The network hub 207 and/or the network switch 209 may be coupled to thenetwork router 211 for connection to the cloud 204. The network router211 works in the network layer of the OSI model. The network router 211creates a route for transmitting data packets received from the networkhub 207 and/or network switch 211 to cloud-based computer resources forfurther processing and manipulation of the data collected by any one ofor all the devices 1 a-1 n/2 a-2 m. The network router 211 may beemployed to connect two or more different networks located in differentlocations, such as, for example, different operating theaters of thesame healthcare facility or different networks located in differentoperating theaters of different healthcare facilities. The networkrouter 211 may send data in the form of packets to the cloud 204 andworks in full duplex mode. Multiple devices can send data at the sametime. The network router 211 uses IP addresses to transfer data.

In an example, the network hub 207 may be implemented as a USB hub,which allows multiple USB devices to be connected to a host computer.The USB hub may expand a single USB port into several tiers so thatthere are more ports available to connect devices to the host systemcomputer. The network hub 207 may include wired or wireless capabilitiesto receive information over a wired channel or a wireless channel. Inone aspect, a wireless USB short-range, high-bandwidth wireless radiocommunication protocol may be employed for communication between thedevices 1 a-1 n and devices 2 a-2 m located in the operating theater.

In examples, the operating theater devices 1 a-1 n/2 a-2 m maycommunicate to the modular communication hub 203 via Bluetooth wirelesstechnology standard for exchanging data over short distances (usingshort-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz)from fixed and mobile devices and building personal area networks(PANs). The operating theater devices 1 a-1 n/2 a-2 m may communicate tothe modular communication hub 203 via a number of wireless or wiredcommunication standards or protocols, including but not limited to Wi-Fi(IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio(NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE,GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, as wellas any other wireless and wired protocols that are designated as 3G, 4G,5G, and beyond. The computing module may include a plurality ofcommunication modules. For instance, a first communication module may bededicated to shorter-range wireless communications such as Wi-Fi andBluetooth, and a second communication module may be dedicated tolonger-range wireless communications such as GPS, EDGE, GPRS, CDMA,WiMAX, LTE, Ev-DO, and others.

The modular communication hub 203 may serve as a central connection forone or all of the operating theater devices 1 a-1 n/2 a-2 m and mayhandle a data type known as frames. Frames may carry the data generatedby the devices 1 a-1 n/2 a-2 m. When a frame is received by the modularcommunication hub 203, it is amplified and transmitted to the networkrouter 211, which transfers the data to the cloud computing resources byusing a number of wireless or wired communication standards orprotocols, as described herein.

The modular communication hub 203 can be used as a standalone device orbe connected to compatible network hubs and network switches to form alarger network. The modular communication hub 203 can be generally easyto install, configure, and maintain, making it a good option fornetworking the operating theater devices 1 a-1 n/2 a-2 m.

FIG. 5 illustrates a computer-implemented interactive surgical system200. The computer-implemented interactive surgical system 200 is similarin many respects to the computer-implemented interactive surgical system100. For example, the computer-implemented interactive surgical system200 includes one or more surgical systems 202, which are similar in manyrespects to the surgical systems 102. Each surgical system 202 includesat least one surgical hub 206 in communication with a cloud 204 that mayinclude a remote server 213. In one aspect, the computer-implementedinteractive surgical system 200 comprises a modular control tower 236connected to multiple operating theater devices such as, for example,intelligent surgical instruments, robots, and other computerized deviceslocated in the operating theater. As shown in FIG. 6, the modularcontrol tower 236 comprises a modular communication hub 203 coupled to acomputer system 210.

As illustrated in the example of FIG. 5, the modular control tower 236may be coupled to an imaging module 238 that may be coupled to anendoscope 239, a generator module 240 that may be coupled to an energydevice 241, a smoke evacuator module 226, a suction/irrigation module228, a communication module 230, a processor module 232, a storage array234, a smart device/instrument 235 optionally coupled to a display 237,and a non-contact sensor module 242. The operating theater devices maybe coupled to cloud computing resources and data storage via the modularcontrol tower 236. A robot hub 222 also may be connected to the modularcontrol tower 236 and to the cloud computing resources. Thedevices/instruments 235, visualization systems 208, among others, may becoupled to the modular control tower 236 via wired or wirelesscommunication standards or protocols, as described herein. The modularcontrol tower 236 may be coupled to a hub display 215 (e.g., monitor,screen) to display and overlay images received from the imaging module,device/instrument display, and/or other visualization systems 208. Thehub display also may display data received from devices connected to themodular control tower in conjunction with images and overlaid images.

FIG. 6 illustrates a surgical hub 206 comprising a plurality of modulescoupled to the modular control tower 236. The modular control tower 236may comprise a modular communication hub 203, e.g., a networkconnectivity device, and a computer system 210 to provide localprocessing, visualization, and imaging, for example. As shown in FIG. 6,the modular communication hub 203 may be connected in a tieredconfiguration to expand the number of modules (e.g., devices) that maybe connected to the modular communication hub 203 and transfer dataassociated with the modules to the computer system 210, cloud computingresources, or both. As shown in FIG. 6, each of the networkhubs/switches in the modular communication hub 203 may include threedownstream ports and one upstream port. The upstream network hub/switchmay be connected to a processor to provide a communication connection tothe cloud computing resources and a local display 217. Communication tothe cloud 204 may be made either through a wired or a wirelesscommunication channel.

The surgical hub 206 may employ a non-contact sensor module 242 tomeasure the dimensions of the operating theater and generate a map ofthe surgical theater using either ultrasonic or laser-type non-contactmeasurement devices. An ultrasound-based non-contact sensor module mayscan the operating theater by transmitting a burst of ultrasound andreceiving the echo when it bounces off the perimeter walls of anoperating theater as described under the heading “Surgical Hub SpatialAwareness Within an Operating Room” in U.S. Patent ApplicationPublication No. US 2019-0200844 A1 (U.S. patent application Ser. No.16/209,385), titled METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY, filed Dec. 4, 2018, which is herein incorporated by referencein its entirety, in which the sensor module is configured to determinethe size of the operating theater and to adjust Bluetooth-pairingdistance limits. A laser-based non-contact sensor module may scan theoperating theater by transmitting laser light pulses, receiving laserlight pulses that bounce off the perimeter walls of the operatingtheater, and comparing the phase of the transmitted pulse to thereceived pulse to determine the size of the operating theater and toadjust Bluetooth pairing distance limits, for example.

The computer system 210 may comprise a processor 244 and a networkinterface 245. The processor 244 can be coupled to a communicationmodule 247, storage 248, memory 249, non-volatile memory 250, andinput/output interface 251 via a system bus. The system bus can be anyof several types of bus structure(s) including the memory bus or memorycontroller, a peripheral bus or external bus, and/or a local bus usingany variety of available bus architectures including, but not limitedto, 9-bit bus, Industrial Standard Architecture (ISA), Micro-CharmelArchitecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics(IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI),USB, Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), Small Computer Systems Interface(SCSI), or any other proprietary bus.

The processor 244 may be any single-core or multicore processor such asthose known under the trade name ARM Cortex by Texas Instruments. In oneaspect, the processor may be an LM4F230H5QR ARM Cortex-M4F ProcessorCore, available from Texas Instruments, for example, comprising anon-chip memory of 256 KB single-cycle flash memory, or othernon-volatile memory, up to 40 MHz, a prefetch buffer to improveperformance above 40 MHz, a 32 KB single-cycle serial random accessmemory (SRAM), an internal read-only memory (ROM) loaded withStellarisWare® software, a 2 KB electrically erasable programmableread-only memory (EEPROM), and/or one or more pulse width modulation(PWM) modules, one or more quadrature encoder inputs (QEI) analogs, oneor more 12-bit analog-to-digital converters (ADCs) with 12 analog inputchannels, details of which are available for the product datasheet.

In one aspect, the processor 244 may comprise a safety controllercomprising two controller-based families such as TMS570 and RM4x, knownunder the trade name Hercules ARM Cortex R4, also by Texas Instruments.The safety controller may be configured specifically for IEC 61508 andISO 26262 safety critical applications, among others, to provideadvanced integrated safety features while delivering scalableperformance, connectivity, and memory options.

The system memory may include volatile memory and non-volatile memory.The basic input/output system (BIOS), containing the basic routines totransfer information between elements within the computer system, suchas during start-up, is stored in non-volatile memory. For example, thenon-volatile memory can include ROM, programmable ROM (PROM),electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatilememory includes random-access memory (RAM), which acts as external cachememory. Moreover, RAM is available in many forms such as SRAM, dynamicRAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and directRambus RAM (DRRAM).

The computer system 210 also may include removable/non-removable,volatile/non-volatile computer storage media, such as for example diskstorage. The disk storage can include, but is not limited to, deviceslike a magnetic disk drive, floppy disk drive, tape drive, Jaz drive,Zip drive, LS-60 drive, flash memory card, or memory stick. In addition,the disk storage can include storage media separately or in combinationwith other storage media including, but not limited to, an optical discdrive such as a compact disc ROM device (CD-ROM), compact discrecordable drive (CD-R Drive), compact disc rewritable drive (CD-RWDrive), or a digital versatile disc ROM drive (DVD-ROM). To facilitatethe connection of the disk storage devices to the system bus, aremovable or non-removable interface may be employed.

It is to be appreciated that the computer system 210 may includesoftware that acts as an intermediary between users and the basiccomputer resources described in a suitable operating environment. Suchsoftware may include an operating system. The operating system, whichcan be stored on the disk storage, may act to control and allocateresources of the computer system. System applications may take advantageof the management of resources by the operating system through programmodules and program data stored either in the system memory or on thedisk storage. It is to be appreciated that various components describedherein can be implemented with various operating systems or combinationsof operating systems.

A user may enter commands or information into the computer system 210through input device(s) coupled to the I/O interface 251. The inputdevices may include, but are not limited to, a pointing device such as amouse, trackball, stylus, touch pad, keyboard, microphone, joystick,game pad, satellite dish, scanner, TV tuner card, digital camera,digital video camera, web camera, and the like. These and other inputdevices connect to the processor through the system bus via interfaceport(s). The interface port(s) include, for example, a serial port, aparallel port, a game port, and a USB. The output device(s) use some ofthe same types of ports as input device(s). Thus, for example, a USBport may be used to provide input to the computer system and to outputinformation from the computer system to an output device. An outputadapter may be provided to illustrate that there can be some outputdevices like monitors, displays, speakers, and printers, among otheroutput devices that may require special adapters. The output adaptersmay include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device andthe system bus. It should be noted that other devices and/or systems ofdevices, such as remote computer(s), may provide both input and outputcapabilities.

The computer system 210 can operate in a networked environment usinglogical connections to one or more remote computers, such as cloudcomputer(s), or local computers. The remote cloud computer(s) can be apersonal computer, server, router, network PC, workstation,microprocessor-based appliance, peer device, or other common networknode, and the like, and typically includes many or all of the elementsdescribed relative to the computer system. For purposes of brevity, onlya memory storage device is illustrated with the remote computer(s). Theremote computer(s) may be logically connected to the computer systemthrough a network interface and then physically connected via acommunication connection. The network interface may encompasscommunication networks such as local area networks (LANs) and wide areanetworks (WANs). LAN technologies may include Fiber Distributed DataInterface (FDDI), Copper Distributed Data Interface (CDDI),Ethernet/IEEE 802.3, Token Ring/IEEE 802.5 and the like. WANtechnologies may include, but are not limited to, point-to-point links,circuit-switching networks like Integrated Services Digital Networks(ISDN) and variations thereon, packet-switching networks, and DigitalSubscriber Lines (DSL).

In various aspects, the computer system 210 of FIG. 6, the imagingmodule 238 and/or visualization system 208, and/or the processor module232 of FIGS. 5-6, may comprise an image processor, image-processingengine, media processor, or any specialized digital signal processor(DSP) used for the processing of digital images. The image processor mayemploy parallel computing with single instruction, multiple data (SIMD)or multiple instruction, multiple data (MIMD) technologies to increasespeed and efficiency. The digital image-processing engine can perform arange of tasks. The image processor may be a system on a chip withmulticore processor architecture.

The communication connection(s) may refer to the hardware/softwareemployed to connect the network interface to the bus. While thecommunication connection is shown for illustrative clarity inside thecomputer system, it can also be external to the computer system 210. Thehardware/software necessary for connection to the network interface mayinclude, for illustrative purposes only, internal and externaltechnologies such as modems, including regular telephone-grade modems,cable modems, and DSL modems, ISDN adapters, and Ethernet cards.

FIG. 7 is a block diagram of an example surgical simulator system. Thesurgical simulator system may include a simulation device 30000. Thesurgical simulator system may include an application creation device30002, a human interface device 30004, a surgeon agent device 30006,and/or a surgical data system 30008.

The simulation device 30000 may provide core simulation functionality.For example, the loading/running of one or more simulations, thereception and processing of user control information input, thegeneration and transmission of visual, audible, and/or hapticinformation output, the collection of simulation operation and activityinformation, and the primary simulation cycle processing may beperformed by the simulation device 30000.

The application creation device 30002 may provide simulation authoringfunctionality. Individual simulation applications may be stored asapplication modules 30010 at the simulation device 30000. Theapplication modules 30010 may be created, modified, and/or deleted bythe application creation device 30002. The application modules 30010 mayinclude computer readable and/or executable instructions to direct anoperation of the simulation device 30000. For example, the applicationmodules 30010 may include any filetype suitable for storing informationto run a surgical simulation, for example, simulation scripts,programming code, structure data files such as Extensible MarkupLanguage (XML) files, database files, and the like.

The application creation device 30002 may include a graphical userinterface with controls to author application modules 30010. Theapplication creation device 3002 may communicate with the simulationdevice 30000 to retrieve, modify, and/or load application modules 30010for simulation operation. For example, the graphical user interface mayinclude interface structures to allow a user to select simulationactivities, to input various simulation parameters, to set simulationobjectives, and to confirm simulation execution. The applicationcreation device 30002 may be provided as a stand-alone device and/orintegrated with one or more other devices of the surgical simulationsystem, such as integrated with the simulation device 30000 for example.

The human interface device 30004 may include any hardware, software,and/or combination thereof that enables a human user to interact with asimulation provided by the simulation device 30000. The human interfacedevice 30004 may enable a user to provide control input to thesimulation device 300000 and/or to receive output information (such asvisual, audible, and/or haptic information) from the simulation device30000. In one example, the human interface device 30004 may include atraditional desktop computer.

The human interface device 30004 may include suitable physicalequipment. For example, the human interface device 30004 may includephysical equipment that mimic physically and/or virtually aspects of asurgical procedure. For example, such equipment may include bench-topunits, part-task virtual reality units, high fidelity virtual realityunits, high fidelity full-size patient units, suite units, high fidelityfull operating room units, full physics virtual reality units, surgicalrobot console units, and the like. For example, the human interfacedevice 30004 may include devices such as the computer-based simulatorinterfaces disclosed by Gallager et al, “Simulations for ProceduralTraining,” Fundamentals of Surgical Simulation, Principles and Practice,Springer (2012).

The human interface device 30004 may include physical equipment thatmimics, physically and/or virtually, surgical instruments. For example,the human interface device 30004 may include physical devices that mimicsurgical instruments, appliances, and consumables, such as accessequipment, such as trocars, hand-access ports, insufflation needles, andguiding sheaths; adjunctive hemostats, such as patches, gelatins, andpowders; craniomaxillofacial appliances, like distractors and plates;balloons and inflators; catheters, like diagnostic catheters, accesscatheters, vascular catheters, and therapeutic catheters; energy sealingand dissecting devices, like tissue sealers, shears, blades, andforceps; orthopedic equipment, like reduction wires, compression screws,plates, implants, drills, burrs, rods, and connectors; ligationinstruments, like open and endoscopic clip appliers; microwave ablationequipment; ancillary endoscopic instruments, like drains, sutures,ligature, needle holders, retrievers, and suture clips; surgicalstapling equipment, like open staplers, endoscopic staplers, cutterstaplers, powered staplers, circular staplers, vascular staplers, linearstaplers, staple cartridges, and staple line reinforcement applicators;wound closure materials, like suture, adhesives, needles, and knotlesstissue control devices; imaging devices, like minimally invasive imagingdevices; and the like. For example, the human interface device 30004 mayinclude virtual reality handheld controllers, that when operated with avirtual reality headset, mimics the surgical instruments, appliances,and consumables, such as those disclosed above.

The human interface device 30004 may include a display that communicatesvisual representations of the simulation to the user. The humaninterface device 30004 may include a computer display. The humaninterface device 30004 may include a virtual reality headset display.For example, the virtual reality headset display may be used display thesurgical environment, such as that disclosed in FIG. 2, herein. A userwith such a virtual reality headset display may view and/or interactwith any of the elements in the surgical operating room 116, including,for example, the patient, the robotic system 110, the surgeon's console118, the surgical robotic hub 122, one or more surgical tools 117, theimaging device 124, the patient side cart 120, one or more displays 119,107, 109, and the like.

The human interface device 30006 may present visual information thatrepresents the point of the view of the surgeon. The human interfacedevice 30006 may present visual information from a simulated imagingdevice, such as an arthroscope, angioscope, bronchoscope,choledochoscope, colonoscope, cytoscope, duodenoscope, enteroscope,esophagogastro-duodenoscope (gastroscope), endoscope, laryngoscope,nasopharyngo-neproscope, sigmoidoscope, thoracoscope, ureteroscope, andtheir related instruments, controls, and the like. The human interfacedevice 30006 may present visual information from a simulatedsupplemental intra-operative imaging equipment, like computed tomography(CT) units, magnetic resonance imaging (MRI) units, image-guided surgeryunits, intra-operative ultrasound units; fluoroscopy units, and thelike. Such point-of-view visual information, surgical imaginginformation, and supplemental intra-operative imaging information may bedisplayed in any combination to the user suitable for the simulation'soperation. For example, such information may be presented to the user asa single full-screen view, a tiled window view, a picture-in-a-pictureview, or registered to a simulated display unit in a virtual realityview.

The human interface device 30004 may include a physical and/or virtualreality surgical robot surgeon console. For example, an examplesurgeon-console-like human interface device 30004 may include a display,such as a stereo vision display and control inputs, including hand-heldmanipulators, foot pedals, and the like. For example, thesurgeon-console-like human interface device 30004 may include aninterface of the surgeon's console 118, disclosed herein. The humaninterface device 30004 may enable voice controls via, for example, amicrophone and speech recognition functionality. The human interfacedevice 30004 may provide audible feedback via, for example, a speaker.The human interface device 30004 may provide haptic feedback via, forexample, vibration, force feedback, air vortex rings, and ultrasoundtechniques.

As implemented, the human interface device 30004 may be provided as astand-alone device and/or integrated with one or more other devices ofthe surgical simulation system, such as integrated with the simulationdevice 30000 for example. The simulation device 30000 may include aninterface module 30012 to communicate with the human interface device30004. In an example, human interface device 30004 may be integratedinto one or more elements of the computer-implemented interactivesurgical system 100. For example, the human interface device 30004 maybe integrated into the computer system 210. For example, the humaninterface device 30004 may be integrated into the hub 106. For example,the human interface device 30004 may be integrated into thevisualization system 108. The interface module 30012 may communicatewith the one or more elements of the computer-implemented interactivesurgical system 100 via the surgical data system interface module 30014for example.

In an embodiment, more than one human interface device 30004 mayconcurrently engage with the simulation device 30000. For example, amulti-person simulation applicatio.

The surgeon agent device 30006 may include any hardware and/or softwaresuitable for providing a computer-based control and response to theinput and output of the simulation device 30000. The surgeon agentdevice 30006 may include a computer process that mimics human input tothe simulation device 30000. For example, the surgeon agent device 30006may be able to record and register control inputs, such as basicinstrument manipulation. The surgeon agent device 30006 may include acomputer process that can access a input/output application programminginterface (API) of the simulation device 30000. For example, the API mayreveal one or more input/output functions that may be directed accordingto the surgeon agent device 3006. The functions may include granularmanipulation and physics-based input/output functions, such as functionsthat directly control the location and movement of instruments. Thefunctions may include less granular surgical-activity-based input/outputfunctions, such as a ligation activity, a suturing activity, a staplingactivity, and the like. The functions may include less granular surgicaltask and/or stage-based input/output functions, such as surgical accessfunction, organ mobilization function, and the like. Each function mayinclude parameter consistent with its level of granularity. Theparameters may provide specific details to direct the operation of thefunction within the simulation. The surgeon agent 30006 may includefunctionality for generating and operating multiple simulation runs. Forexample, a user may wish to estimate the duration of various suturingtechniques. A surgeon agent device 30006 may be used to script thesimulation of any number of different techniques, each of which can berun via the simulation device, and the metrics collected by thesimulation device may be used to estimate the difference in durations.

The surgeon agent device 30006 may be provided as a stand along deviceand/or integrated with one or more other devices of the surgicalsimulation system, such as integrated with the simulation device 30000for example. The simulation device 30000 may include an interface module30012 to communicate with the surgeon agent device 30006. For example,the surgeon agent device 30006 may be integrated as a module of thesimulation device 30000. For example, the surgeon agent device 30006 maybe integrated into an application module 30010 of the simulation device.

The surgical data system 30008 may include any hardware and/or softwaresuitable for providing external, structured surgical information andfunctionality to the simulation device 30000. The surgical data system30008 may include the structure and/or functions described in connectionwith FIGS. 1-6 herein. For example, the surgical data system 30008 mayinclude one or more elements of a computer-implemented interactivesurgical system 100. The surgical data system 30008 may include, forexample, a surgical hub 106. For example, the simulation device 30000include a surgical data system interface module 30014 that enablescommunication with the surgical hub 106 via the surgical hub'scommunication module 130. The surgical data system 30008 may include,for example, on or more surgical data repositories. For example, thesurgical data system 30008 may include the computer system 210 locatedin the surgical theater. For example, the surgical data system 30008 mayinclude the remote server 213 in the cloud 204.

A surgical data system 30008, such as the surgical hub 106 for example,may provide data to the simulation device 30000 and/or the applicationcreation device 30002. For example, the data may include any surgicaldata collected and/or generated by the surgical hub 106. Also forexample, the simulation device 30000 may receive similar data directlyfrom any of the networked devices disclosed in FIGS. 1-6. Such data mayinclude information about a live surgical procedure, for example. Suchdata may include information about a past surgical procedure. Such datamay include information about future, scheduled surgical procedures.

Information about the surgical procedures may include information aboutthe patient, the staff, the procedure as planned, the procedure asexperienced, and post-operative activity including patient outcomes. Forexample, the information received and used by the simulation device mayinclude patient records, patient imaging, models of patient anatomy,patient lab results, patient medical history, and the like. For example,the information received and used by the simulation device may include astaff manifest for a procedure, details about the past procedures of thespecific staff members, staff metrics, experience, recent scheduling andworkload, and historical surgical activity, such instrument usestatistics, procedure duration, and the like. For example, theinformation received and used by the simulation device may includeprocedure plans, equipment and inventory information, pull-lists,checklists, procedure plan analysis and recommendations. For example,the information received and used by the simulation device may includeany data collected or generated during a live procedure, such asprocedure progress, milestones, patient information, vitals, operatingtheater setup, staff movement, imaging, instrument use, surgicaltechnique, such as that captured by video, recorded manually, and/orinferred from smart-instrument reporting for example, duration, abnormalevent reporting, and the like. Any data captured during a live proceduremay also be stored and made available as a past procedure. For example,the information received and used by the simulation device may includepost-operative records, patient recovery information, and patientoutcome information, post-operative diagnostic information, such aslabs, imaging, etc.

The simulation device 30000 may include any computer or processingplatform suitable for executing one or more simulations. The simulationmay include a computer-modeled environment of a surgical procedure. Forexample, the simulation may include a model of a patient's anatomyand/or physiology. For example, the simulation may include a model ofthe actions and/or instruments of one or more healthcare professionals,such as the actions of a surgeon, nurse, other doctor, technician, orthe like.

The simulation device 30000 may include one or more functional modules.Each module may include hardware, software, or a combination thereofthat enable functionality of the module. One or more modules, operatingin concert, may represent a computer framework on which a simulation ofa medical procedure may be executed. The modules may include hardwareelements, such as a computer processing unit, a graphics processingunit, a field-programmable gate array (FPGAs), communications hardware,memory, and the like. The modules may include software elements thatwhen executed by a processor cause the module to perform certainfunctions.

The simulation device may include a core simulation module 30016, asimulation applications module directory 30018, the interface module30012, an object properties module 30020, a physics module 30022, aphysiology model 30024, a texture model 30026, a 3D graphics pipeline30028, the surgical data system interface module 30014, a metricsextraction module 30030, a session storage and management module 30032,for example. The simulation device may include an operating systemmodule 30034.

The core simulation model 30016 may provide primary simulationfunctionality of the simulation device 30000. For example, the coresimulation module 30016 may include code for initializing a simulation,for communicating and interacting with other modules of the simulationdevice 30000, and/or for managing architectural level simulationparameters. For example, the core simulation module 30016 may include amaster event clock to provide time alignment and/or coordination of theoperation of the modules of the simulation device 30000. For example,the core simulation module 30016 may establish the overall simulationframe rate.

The core simulation module 30016 may include core for providing a mastersimulation cycle. The core simulation module 30016 may run one or moreiteration of the master simulation cycle. Each iteration of the mastersimulation cycle may represent an individual time slice for simulation.In an example, the core simulation module 30016 may run the mastersimulation cycle according to the flow disclosed in FIG. 10.

The simulation applications module directory 30018 may manage thestoring, retrieving, and/or linking of the one or more applicationmodules 30010. Each application module 30010 may include code thatdirects the application-level aspects of a simulation. For example, anapplication module 30010 may include the functionality to provide asimulation of specific anatomy, of specific teaching scope, of specificequipment, or the like. In an example simulation device 30000, anapplication-specific simulation device 30000 may operate with a singleapplication module 30010 with or without a simulation application moduledirectory 30010. The simulation application module directory 30018 mayoperate based on interaction with the core simulation module 30016and/or the application creation device 30002.

The interface module 30012 may provide functionality for interactingwith the human interface device 30004 and/or the surgeon agent device30006. For example, the interface module 30012 may include one or moredrivers to translate information received from human interface device30004 into software commands, interrupts, and the like. For example, theinterface module 30012 may include a software application programminginterface (API) for interacting with the surgeon agent 30006. Theinterface module 30012 may provide information received from the humaninterface module 30004 and/or the surgeon agent device 30006 to othermodules of the simulation device 30000. For example, the interfacemodule 30012 may receive a control input from the human interface module30004 and/or the surgeon agent device 30006 that represents movement ofa simulated instrument and provide that information to one or more othermodules of the simulation device 30000 so the movement may berepresented in the simulation.

The interface module 30012 may provide the API to enable a more granularinteraction with the surgeon agent device 30006. For example, the APImay provide an interface to receive simulation parameters and simulationsettings from the surgeon agent device 30006. Such simulation parametersand/or simulation settings may be like those input by the user via theapplication creation device 30002, for example. For example, the surgeonagent device 30006 may be enabled to run one or more computer-controlledsimulation trials through the simulation device 30000. For example, thesurgeon agent device 30006 may be enabled to run multiple simulations,each with alternative interactions.

The interface module 30012 may send output from the simulation device30000 to the human interface device 30004 and/or the surgeon agentdevice 30006. For example, the output may include visual output, hapticoutput, audio output, and/or structured data output, or the like.

The object properties module 30020 may provide functionality formanaging the simulated appearance and/or behavior of objects within inthe simulation. Simulated objects may include objects such as anatomy,instrument, equipment, consumables, fluids, and the like. An object'sappearance may be managed by object properties, such as location,dimensions, scale, material, parent/child relationships, vertices,faces, interactivity, transparency, trajectory, rendering properties,textures, surface reflectivity, motion blur, layering, and the like. Anobject's behavior may be managed by object properties, such as physicsproperties, mass, motion, collision behavior, elasticity, viscosity,surface tension, rigging constraints, hardness, shear strength, tearingbehavior, grain, and the like. The physics module 30022 may providefunctionality to calculate the physical responses and/or interaction ofobjects within the simulation. The physical module may determine suchresponses and/or interactions according to classical mechanics, fluidmechanics, soft body dynamics, Brownian motion, collision detection,cloth behavior, finite element analysis, and the like. The physicsmodule 30022 may include commercial and/or open-source modules, such asPhysX™, Simulation Open Framework Architecture (SOFA)™, VisSim™, and thelike.

The physiology module 30024 may provide functionality to calculatephysiological responses and/or interactions of the anatomy and/orpatient as a whole in the simulation. The physiology module 30024 mayprovide physiological models for key organs and/or systems. Thephysiological models may include mathematical models, statisticalmodels, or the like. For example, the physiology module 30024 may modulethe patient's vitals to calculate their response and/or interaction toactivities performed during the simulation. For example, a circulatorymodel may calculate blood pressure in response to a severed vessel inthe simulation. The physiology module 30024 and the physics module 30022may coordinate with each other during the calculation of each state ofthe simulation. For example, blood pressure calculated by thecirculatory model may be used to determine fluid dynamics propertiescalculated by the physics module 30022 and managed by the objectproperties module 30020.

The texture module 30026 may provide functionality to determine,retrieve, and/or generate the appropriate surfacing of objects withinthe simulation. The texture module 30026 may include one or moresurfacing modalities that may be controlled according to parameters ofthe simulation. The surfacing modalities may include artificiallygenerated surfaces, surfaces based on real-world imagery, andcombinations thereof. The texture module 30026 may coordinate operationwith the physics module 30022 to provide accurate haptic feedback to theuser via the user interface module 30012.

The 3D graphics pipeline 30028 may provide functionality for visualrendering of the simulation environment. The 3D graphics pipeline 30028may receive object properties and a perspective. The 3D graphicspipeline 30028 may determine the visualization to be presented to theuser that represents the objects in 3D space as viewed from the cameraperspective. The 3D graphics pipeline 30028 may determine geometricaspects of the rendering, such as lighting, projection, clipping, viewtransformation, and the like. The 3D graphics pipeline 30028 maydetermine rasterization aspects of the rendering, such as fragmentation,pixel shading, vertex shading, geometry sharing, texture filtering, andthe like. The 3D graphics pipeline 30028 may coordinate with the texturemodule 30026 to provide accurate visual feedback to the user via theinterface module 30012.

The surgical data system interface module 30014 may provide interactiveconnectivity to one or more elements of computer-implemented interactivesurgical system 100. Information from the one or more elements of thecomputer-implemented interactive surgical system 100 may be communicatedvia the surgical data system interface module 30014 to one more modulesof the simulation device 30000 to influence operation of a simulation.For example, the surgical data system interface module 30014 may receiveinformation about a surgical procedure an communicate it to acorresponding application module 30010. For example, the surgical datasystem interface module 30014 may receive information about aninstrument and communicate it to the object properties module 30020. Forexample, the surgical data system interface module 30014 may receiveinformation about a patient and communicate to the physiology module.For example, the surgical data system interface module 30014 may receiveinformation about tissue imaging and communicate it to the texturemodule 30026.

Information from the modules of the simulation device 30000 may beprovided, via the surgical data system interface 30014, to one or moreelements of the computer-implemented interactive surgical system 100.For example, one or more elements of the computer-implementedinteractive surgical system 100 may receive statistics related to asimulated procedure plan from the metrics extraction module 30030. Forexample, one or more elements of the computer-implemented interactivesurgical system 100 may receive replayed simulation visualizationprocedure plan from the session storage and management module 30032. Forexample, the surgical data system interface module 30014 may provide acommunications pathway between the interface module 30012 and one ormore elements of the computer-implemented interactive surgical system100. For example, a surgeon during a live surgical procedure may accesssimulation information and/or operate a simulation from the operatingtheater. For example, a surgeon may use the surgeon console 118 toaccess and/or interact with a simulation that corresponds to the livesurgical procedure.

The metrics extraction module 30014 may provide recording functionalityof various parameters related to the operation of the simulation. Forexample, the metrics extraction module 30014 may record metrics relatedto the simulation as a whole, such as duration, number of activities,number of movements, complexity of movements, staff employed, staffmovement, equipment and/or instrument changes, etc. For example, themetrics extraction module 30014 may record metrics related to aparticular aspect of the simulation, such as simulated patient vitals,complications, collisions, bleeding, etc. The metrics extraction module30014 may maintain a master log of metric-related events during asimulation. For metrics extraction module 30014 may recordmetric-related events according to a configuration from the applicationmodule 30010 employed for the simulation.

The session storage and management module 30032 may provide managementfunctionality of the main simulation run-record. For example, thesession storage and management module 30032 may store the information toenable a simulation to be rerun, viewed, and/or analyzed in itsentirety. The session storage and management module 30032 may store theinformation about each input, simulation state, and output, such as theinput, simulation state, and output disclosed with regard to FIG. 10.The session storage and management module 30032 may enable a previoussimulation to be recalled, copied, and initialized with new user input.To illustrate, a surgeon in training may recall a simulation run by anexperienced surgeon, pause the simulation at a critical step, andattempt that step on her own. The session storage and management module30032 may provide overlay functionality between various runs of aparticular simulation. Such overlays may highlight similarities anddifferences and may enhance training.

The operating system module 30034 may manage the hardware and/orsoftware resources for the simulation device 30000. The operating systemmodule 30034 may provide common computing system-level services for theother modules of simulation device 30000. For example, the operatingsystem module 30034 may provide hardware input and output handling,memory allocation, hardware interrupt handling, software interrupthandling, thread processing, single task handling, multi-task handling,and the like. The simulation device 30000 may be a real-time computingdevice. The operating system module 30034 may include a real-timeoperating system. For example, the operating system module 30034 may bedriven by the events and frame rate established by the core simulationmodule 30016.

FIG. 8 is a block diagram of an example surgical simulator system. Thesimulation device 30000 is depicted with an example hardwarearchitecture. For example, the simulation device 30000 may include aprocessor 30034, a memory 30036, a storage 30038, a display adapter30040, a manipulation interface adapter 30042, a surgical data systemadapter 30044, and/or a network adapter 30046. One or more of theprocessor 30034, a memory 30036, a storage 30038, a display adapter30040, a manipulation interface adapter 30042, a surgical data systemadapter 30044, and/or a network adapter 30046 may be used to enableoperation of the modules of the simulation device 30000 disclosedherein.

The processor 30046 may include computer processing unit, graphicsprocessing unit, any suitable microcontroller, microprocessor, fieldprogrammable gate array (FPGA), application specific integrated circuit(ASIC), or the like, and/or any combination thereof that is suitable forprocessing and delivering a 3D simulated environment for interactionwith a computer agent and/or human user. In one example, the processor30046 may include one or more processing units. The processor 30046 maybe a processor of any suitable depth to perform the digital processingrequirements disclosed herein. For example, the processor 30046 a 32-bitprocessor, a 64-bit processor, a 128-bit processor, or the like.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The memory 30036 may include any component or collection of componentssuitable for storing data. For example, the memory 30036 may includevolatile memory and/or nonvolatile memory. The memory 30036 may includerandom-access memory (RAM), read-only memory (ROM), erasableprogrammable read-only memory (EPROM), (electrically erasableprogrammable read-only memory) EEPROM, flash memory, or the like.

The storage 30038 may include any component or collection of componentssuitable for storing large quantities of data. For example, storage30038 may include hard disk drives (HDD), solid state drives (SSD),network-attached storage (NAS), or the like. The storage 30038 mayinclude a database structure and/or a database management system (DBMS).

The display adapter 30040 may include any component or collection ofcomponents suitable for outputting the visual representation of a 3Dsimulation environment. For example, the display adapter 30040 mayinclude a graphics card, a display card, a graphics adapter, or thelike. The display adapter 30040 may be used to generates a feed ofoutput images to a display device, such as a display of the humaninterface device 30004. The display adapter 30040 may include a graphicsprocessing unit (GPU). The display adapter 30040 may include hardware torender a graphics pipeline, for example. The manipulation interfaceadapter 30042 may include any component or collection of componentssuitable for receiving manipulation information from the human interfacedevice and/or outputting feedback information to the human interfacedevice. For example, the manipulation interface adapter 30042 mayreceive motion tracking information from a virtual reality headset andin turn, manipulate the view being displayed to the user. For example,the manipulation interface adapter 30042 may receive control inputindicative of a user manipulating a surgical instrument and, in turn,output haptic feedback to the user's handheld device. For example, themanipulation interface adapter 30042 may receive control informationfrom a traditional desktop keyboard and mouse. The manipulationinterface adapter may include input/output hardware such as serialinput/output ports, parallel input/output ports, universal asynchronousreceiver transmitters (UARTs), discrete logic input/output pins,analog-to-digital converters, digital-to-analog converters, universalserial bus (USB) ports, USB-C ports, FireWire ports, High PerformanceParallel Interface (HIPPI), Thunderbolt port, Yapbus, Ethernet, GigabitEthernet, and/or any other suitable peripheral interface technology.

The surgical data system adapter 30044 may include any component orcollection of components suitable for communicating with the surgicaldata system 30008. The surgical data system adapter 30044 may includecommunications hardware to establish a physical channel between thesimulation device 30000 and the surgical data system 30008. For example,the surgical data system adapter 30044 may include a communication portsuch as, a USB port, USB-C ports, FireWire ports, HIPPI port,Thunderbolt port, Yapbus port, Ethernet port, Gigabit Ethernet port,and/or any other suitable peripheral interface. The surgical data systemadapter 30044 may include hardware, software, and/or a combinationthereof to establish a logical channel between the simulation device30000 and the surgical data system 30008 over the network adapter 30046and the network 30048.

The network adapter 30046 may include any component or collection ofcomponents suitable for communication over a network, such as network30048 for example. The network adapter 30046 may enable communicationover networks such as local area networks (LANs), wide area networks(WANs), and/or mobile networks. LAN technologies may include FiberDistributed Data Interface (FDDI), Copper Distributed Data Interface(CDDI), Ethernet/IEEE 802.3, Token Ring/IEEE 802.5, Wi-Fi/IEEE 802.11,and the like. WAN technologies may include, but are not limited to,point-to-point links, circuit-switching networks like IntegratedServices Digital Networks (ISDN) and variations thereon,packet-switching networks, and Digital Subscriber Lines (DSL). Themobile networks may include communication links based on one or more ofthe following mobile communication protocols: GSM/GPRS/EDGE (2G),UMTS/HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A),new radio (NR) or 5G, etc.

In an embodiment, the network adapter 30046 may include a wirelessnetwork adapter, such as a 5G network adapter. Such a 5G network adapter30046 may use a 5G New Radio (NR) transceiver to provide enhanced mobilebroadband (eMBB) with ultra-reliable and low latency communications(URLLC). Such a 5G network adapter 30046 may use wireless bands, such ashigher wireless bands like the 3.5 Ghz-7 Ghz and/or the 24 GHz-48 GHzbands. The network 30048 servicing such a 5G network adapter 30046 mayinclude a public wireless network, a semi-private (e.g., networkslicing-based) network, and/or a fully private wireless network.

FIG. 9 is a block diagram depicting an example surgical simulator humanuser interface device 30004. The human user interface device 30004 isdepicted with an example hardware architecture. For example, the humanuser interface device 30004 may include a processor 30050, a memory30052, a display subsystem 30054, and/or a manipulation subsystem 30056.

The processor 30050 may include computer processing unit, graphicsprocessing unit, any suitable microcontroller, microprocessor, fieldprogrammable gate array (FPGA), application specific integrated circuit(ASIC), or the like, and/or any combination thereof that is suitable forhandling the processing associated with displaying visual informationreceived from the simulation device 30000, processing manipulationinformation for sending to the simulation device, processing feedbackinformation received from the simulation device 30000, and the like. Theprocessor 30050 may include a microcontroller to interface with one ormore local sensors to sense control manipulation from the user and/or tointerface with one or more local actuators to provide feedback from theuser.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The memory 30036 may include any component or collection of componentssuitable for storing data. For example, the memory 30036 may includevolatile memory and/or nonvolatile memory. The memory 30036 may includerandom-access memory (RAM), read-only memory (ROM), erasableprogrammable read-only memory (EPROM), (electrically erasableprogrammable read-only memory) EEPROM, flash memory, or the like.

The display subsystem 30054 may include any component or collection ofcomponents suitable for displaying visual representations of a 3Dsimulation from the simulation device 30000 to a user. The displaysubsystem may include display hardware such as a monitor, a digitalprojector, a smart phone, a digital headset, a virtual reality headset,a stereoscopic display, a robotic surgery surgeon's console display, asurgical display unit, a surgical microscope, and the like.

The manipulation subsystem 30056 may include any component or collectionof components suitable for collecting manipulation controls from theuser to send to the simulation device 30000 and/or providing feedbackinformation, received from the simulation device 30000, to the user.Manipulation from the user may include any interface with sensors thatengage with the user, for example, engaging to indicate a user's intentin the simulation. For example, the interfaces may include keyboards,mice, joysticks, physical equipment that mimics the size, shape, andoperation of actual surgical instruments, virtual reality hand-heldcontrollers, smart gloves, motion sensing systems (such as hand trackingsystems, for example), a robotic surgery surgeon's console manipulatorsand/or controls, a physical unit that mimics the size, shape, andoperation of an actual robotic surgery surgeon's console manipulatorsand/or controls, and the like. For example, the interface may include apoint of view sensor, such as an accelerometer, in a headset to indicatea user's point of view within the simulation.

Feedback from the simulation device 30000 may include any interface withan actuator that provides sensory input to the user. For example, thefeedback may include haptic feedback, force feedback, temperaturefeedback, moisture feedback, audio feedback, olfactory feedback, and thelike. For example, a force feedback and/or haptic actuator in themanipulator of a robotic surgery surgeon's console may be used tosimulate the feedback the user would feel if operating such amanipulator in a live procedure. For example, a force feedback and/orhaptic actuator in a user device that mimics the size, shape, andoperation of actual surgical stapler may be used to simulate thefeedback the user would feel if operating such a device on live tissue,including force feedback when engaging the tissue and firing the staplerfor example.

FIG. 10 is a flow chart of an example surgical simulator operation. At30058, a simulation application may be loaded. For example, the coresimulation module 30016 may cause data associated with a particularapplication module 30010 to be loaded into memory 30036. The loaded datamay include instructions for the processor 30034 to operate a particularsimulation. The loaded data may include a procedural plan for thesimulation. For example, the procedural plan may be structured asdisclosed herein, for example with regard to FIGS. 11A-B. The loadeddata may include an initial state for the simulation.

At 30060, the simulation output may be determined and/or sent. Forexample, the simulation output may be determined and/or sent by thesimulation device 30000. Here, the core simulation module 30016 mayreference a current state of the simulation (e.g., an initial stateand/or a subsequent state). The core simulation module 30016 may engageone or more other modules to process the current state for output. Forexample, the core simulation module may engage any of the objectproperties module 30020, the texture module 30026, the applicationmodule 30010, the 3D graphics pipeline 30028, the interface module30012, and/or the surgical data system interface module 30014 to processthe current simulation state into information for output. Informationrelated to the output maybe processed and/or stored by the metricsextraction module 30030 and/or the session storage and management module30032, for example.

In a human-operated simulation session, for example, output informationmay be sent via the display adapter 30040 and/or the manipulationinterface adapter 30042 to the display subsystem 30054 and/or themanipulation subsystem 30056 of the human interface device 30004. In acomputer-controlled simulation session, for example, output informationmay be sent via the interface module 30012 to a surgeon agent 30006.Also for example, in a computer controlled simulation session, outputinformation may be sent (e.g., processed locally) at an applicationmodule 30010. In a session accessed via the surgical data system 30008,for example, output information may be sent by the surgical data systeminterface module 30014 via the surgical data system adapter 30044 and/orthe network adapter 30046.

At 30062, simulation input may be received and/or processed. Forexample, simulation input may be received and/or processed by thesimulation device 30000. Here, the core simulation module may engagewith the interface device, the surgical data system interface module,and/or the application module 30010 to receive control input.Information related to the input maybe processed and/or stored by themetrics extraction module 30030 and/or the session storage andmanagement module 30032, for example.

In a human-operated simulation session, for example, input informationmay be sent from a manipulation subsystem 30056 of the human interfacedevice 30004 and received via the manipulation interface adapter 30042.In a computer-controlled simulation session, for example, inputinformation may be sent from a surgeon agent 30006 and received via theinterface module 30012. Also for example, in a computer controlledsimulation session, input information may be received (e.g., processedlocally) at an application module 30010. In a session accessed via thesurgical data system 30008, for example, input information may bereceived via the surgical data system adapter 30044 and/or the networkadapter 30046 and initially handled by the surgical data systeminterface module 30014.

At 30064, a subsequent simulation state may be determined. For example,a subsequent simulation state may be determined from the currentsimulation state and/or the any received input. The core simulationmodule 30016 may engage one or more of the other modules of thesimulation device 30000 to determine the subsequent simulation state.For example, the code simulation module 30016 may engage the applicationmodule, the object properties module, the physics module, the physiologymodule, and the like. The subsequent simulation state may be determinedby operation of the processor 30034. Information related to the inputmaybe processed and/or stored by the metrics extraction module 30030and/or the session storage and management module 30032, for example.

At this stage, the process may loop to receiving input at 30060. Eachiteration of this flow may represent a corresponding time cycle in thesimulation. The framerate of the simulation may be set to a levelsuitable for the goal of the simulation and the processing capabilitiesof the surgical simulation device 30000. Lower framerates may enableprocessing that achieves a live human interaction simulation. Higherframerates may enable greater simulation fidelity. For example, whenoperating computer-controlled simulations, with a surgeon agent 30006for example, a higher framerate may be used, even if the higherframerate causes the processing time of the simulation to exceed thereal-world time it is simulating.

FIGS. 11A-B illustrate example surgical procedural plan data structuresfor use with a computer-implemented interactive surgical system and/or asurgical simulator. A surgical procedure plan may include informationthat outlines the staff, equipment, technique, and steps that may beused to perform a surgical procedure. For example, the procedure planmay include a staff manifest indicating what roles and/or what specifichealth care professionals are to be involved in the procedure. Theprocedure plan may include a listing of equipment, such as durablesurgical equipment, imaging equipment, instruments, consumables, etc.that may be used during the procedure. For example, the procedure planmay include a pick list for a surgical technician to use to assemble theappropriate tools and materials for the surgeon and the surgery whenprepping the operating theater. The procedure plan may includeinformation about the procedure's expected technique. For example, theprocedure plans for the same surgical goal may include different methodsof access, mobilization, inspection, tissue joining, wound closure, andthe like.

The procedure plan may reflect a surgeon's professional judgement withregard to an individual case. The procedure plan may reflect a surgeon'spreference for and/or experience with a particular technique. Theprocedure plan may map specific surgical tasks to roles and equipment.The procedure plan may provide an expected timeline for the procedure.

The procedure plan may include one or more decision points and/orbranches. Such decision points and/or branches may provide surgicalalternatives that are available for particular aspects of the procedure,where selection of one of the alternatives may be based on informationfrom the surgery itself. For example, the choice of one or morealternatives may be selected based on the particular planes of theparticular patient's anatomy, and the surgeon may select an alternativebased on her assessment of the patient's tissue during the live surgery.

The procedural plan may include one or more contingencies. These mayinclude information about unlikely but possible situations that mayarise during the live surgery. The contingencies may include one or moresurgical tasks that may be employed if the situation does occur. Thecontingencies may be used to ensure that adequate equipment, staff,and/or consumables are at the ready during the procedure.

The procedure plan may be recorded in one or more data structures. Aprocedure plan data structure may be used to record data about a futurelive surgery, about a completed live surgery, about a future simulatedsurgery, about a completed simulated surgery, and the like. A procedureplan data structure for live surgeries may be used by thecomputer-implemented interactive surgical system 100. For example, theprocedure plan data structure for live surgeries may be used by surgicalhub 106 to enhance situational awareness and/or the operational aspectsof the computer-implemented interactive surgical system 100. Theprocedure plan data structure for live surgeries may be used by thesurgical hub 106 to record discrete elements of the live surgery forstructured analysis.

A procedure plan data structure may be used by a simulation device30000. For example, the procedure plan data structure may be used by thesimulation device 30000 to establish a setting and/or one or moreobjectives for a simulation session. For example, the procedure plandata structure may be used by the simulation device 30000 to record thediscrete elements of the simulated surgery for structured analysis.

The procedure plan data structure may include any structure suitable forcapturing data elements related to the procedure. For example, theprocedure plan may be recorded in a tree-like data structure, such asthe one shown in FIG. 11A, for example. Here, the root of the treestructure represents the core procedure data 30066. The core proceduredata 30066 may include information about the procedure as a whole, suchas procedure name, procedure code, patient name, date, time, and thelike. For a simulation, the core procedure data 30066 may includeinformation about simulation device, such as device ID, softwareversion, user, the simulation run settings, such as frame rate,resolution, connected user interface devices, and the like.

The procedure data may include leaves of the tree structure. The firstlevel of leaves may include data regarding the main aspects of theprocedure plan, such as the procedure setup 30068, one or more procedurestages 30070, one or more contingencies 30072, and the data regardingthe result of the procedure 30074.

The setup data 30068 may include information related to the preparationsand/or initial state of the procedure. For example, the setup data 30068may include elements such as staff manifest, staff roles and/or staffIDs, operating room ID, an equipment list, a room layout, an initialsurgical table position, a listing of instruments and/or consumables onprepared in the surgical field, any initial settings associated withequipment, pre-surgical imaging, patient record, etc. For a simulation,the setup data 30068 may include information related the simulatedenvironment, such as a record of the simulated anatomy, a record of thesimulated physiology, pre-surgical imaging, and the like.

The stage data 30070 may include data elements related to a majormilestone of the procedure. For example, a stage of a procedure mayinclude a milestone such as establishing access. The stage data 30070may include information related to the staff, equipment, technique, andsteps that may be used to perform the particular stage of the procedure.The stage data 30070 may include a stage ID.

The stage may be further detailed by one or more sub-leaves, such as oneor more surgical tasks 30076. The surgical task may represent a discretesurgical step within a given stage. For example, within the stage ofaccess, placing a trocar may be a surgical task. The surgical task data30076 may include a task ID. The surgical task data 30076 may includeinformation related to the particular task, such as the staff and/orsurgeon performing the task, the equipment to be used, the particulartechnique being applied, the patient vitals at the time the task isbeing performed, other environment information, and the list. Each taskmay be further detailed with goal data 30078, data related to ananatomy-instrument interaction 30080, and result data 30082. The goaldata 30078 may include information indicative of the relative success ofthe task performance. The goal data 30078 may include information aboutexpected task duration, acceptable performance specificity, efficiencymodality, avoidance of complications, and the like. The result data30082 may include information related to one or more goals. The resultdata 30082 may record the surgical performance (e.g., live and/orsimulated) relative to the goals.

The task data 30076 may include one or more elements ofanatomy-instrument interaction data 30080. The anatomy-instrumentinteraction data 30080 may represent a granular indication of surgicalperformance. The anatomy-instrument interaction data 30080 may representthe one or more specific activities used to perform the surgical task.The anatomy-instrument interaction data 30080 may represent theobservable behavior of the surgeon.

In an example, the anatomy-instrument interaction data 30080 may includethe specific positions, forces, angles, and the like being applied tothe anatomy by the surgeon. For example in a live surgery, data recordedfrom smart instruments by the surgical hub 106 may be captured asanatomy-instrument interaction data 30080. For example, a smart surgicalstapler in cooperation with other elements of the computer-implementedinteractive surgical system 100 may record stapler position, angle, tipforces, jaw forces, staple cartridge type, closing pressure, firingrate, and the like. In a simulated surgery, similar data elements may becaptured.

The contingency data 30072 may indicate any complications that may berelevant to the procedure. Each contingency data 30072 may include oneor more task data elements 30084 that address the appropriate responseto the particular complication. The contingency data 30072 may indicatedeviations from an original procedure plan. Also for example,contingency data may be cross-referenced to one or more tasks 30078and/or anatomy-instrument interactions 30080. For example, if a certainperformance in an anatomy-instrument interactions 30080 could lead to acomplication, the nature of that performance and a cross-reference tothe contingency may include in the result data 30082 associated withthat anatomy-instrument interactions 30080.

The result data 30074 may be indicative of the result of the procedure.Here overall metrics of the surgical performance may be stored, notes,actual and/or simulated patient recovery information, and/or patientoutcomes. For example, the result data 30074 may include efficiencyinformation, cost information, surgical duration, workload metrics,percentage of planned consumables used, and the like.

FIG. 11B illustrates a procedural plan data structure with the abovedisclosed elements, which further establishes structure of alternativesteps for completing a particular procedure, task, or activity. Asshown, the procedure represented by the procedure data 30086 may includetwo alternative setups, each indicated by respective setup data-a firstsetup data 30088, 30090 and a second setup data 30092. The first setupdata 30088, 30090 may include two alternative tasks 30094, 30096. Thesecond setup data 30092 may include one task 30098. In thisillustration, the procedure represented by procedure data 30086 may beaccomplished in three different ways. First via first setup 30088 andthe first task 30094. Second via the first setup 30090 and the secondtask 30096. And third via the second setup 30092 and its correspondingtask 30098.

Each path of the tree structure may represent a particular set ofalternative ways to perform the procedure. Such a structure may beuseful to aid the creation of a particular procedure plan for aparticular live and/or simulated surgery. Such a structure may be usefulto simulate many possible alternatives of a procedure to assess thedifferences in results.

FIG. 12 illustrates a system for providing simulation support in a livesurgical procedure. A live surgical procedure may employ a simulationdevice 31500, a surgical data system, and the like. There may be storeda simulation of a particular surgical procedure. And a surgeon maybenefit, when performing that particular procedure, from retrieving thestored simulation and interacting with it during the live procedure. Thestored simulation may provide guidance, training, aid in memory and/orrecall, and the like. The stored simulation may enable the surgeon tosimulate a task and/or technique in temporal proximity to performing thetask and/or technique live. The stored simulation may provide expertguidance on a particular task and/or technique, where the simulatedanatomy and/or simulated patient conditions aligns with the livepatient.

The simulation device 31500 may include a computing platform capable ofstoring, recalling, and/or running a simulation as disclosed herein. Forexample, the simulation device 31500 may include simulation device 30000as disclosed herein. The simulation device 31500 may be in communicationwith a surgical data system 31502. For example, the simulation device31500 may be in communication with a surgical data system 31502 via acommunications leg 31504.

The surgical data system 31502 may include a surgical computing platformcapable of providing surgical situational awareness, live imaging,simulation imaging, and the like. For example, the surgical data system31502 may include the surgical data system 30008 disclosed herein. Forexample, the surgical data system 31502 may include one or morecomponents of the computer-implemented interactive surgical system 100disclosed herein. For example, the surgical data system 31502 mayinclude the surgical hub 106 disclosed herein.

The surgical data system 31502 may be in communication with a simulationhuman user interface 31506. For example, the surgical data system 31502may be in communication with the simulation human user interface 31506via a communications leg 31508. The human interface device 31506 mayinclude the human interface device 30004 disclosed herein. In anexample, the human interface device 31506 may include user controlsparticularly suited for the operating room environment. For example, thehuman interface device 31506 may include simplified play-back controls.For example, the human interface device 31506 may include a roboticsurgery surgeon's console with a simulation/live cutover switch, suchthat the same console may be used to interact with the simulation andinteract with the live patient. In an embodiment, such a human interfacedevice 31506 may enable a video overlay of imaging of the live procedureand imaging of the simulation.

The surgical data system 31502 may be in communication with a surgicaldisplay 31510. For example, the surgical data system 31502 may be incommunication with the surgical display 31510 via a communications leg31512. The surgical display 31510 may include one or more components ofthe computer-implemented interactive surgical system 100 disclosedherein. For example, the surgical display 31510 may include aspects ofthe visualization system 108, such as the primary display 119, the firstnon-sterile display 107, the second non-sterile display 109, and thelike.

The surgical data system 31502 and/or the simulation device 31500 may beused to access archived portions of a simulation during the procedure,for example, to reorient and/or reanalyze key steps due to unanticipateddeviations from the pre-surgery plan. In an example, a procedure planmay be created with surgical choices at key surgical steps. A simulationof this procedure plan may be recorded, archived, and/or later recalledduring the actual procedure. For example, such simulated snap shots maybe recalled by the surgeon to review and/or update with new data and/orunanticipated issues encountered during the procedure. For example, thesimulation may be re-run during the procedure to provide the surgeonwith new outcomes, choices, and/or impacts. Also for example, thesimulation may be updated with information learned during the procedure,such as revising the simulated patient's anatomy. Such an updatedsimulation may be rerun during the procedure according to the procedureplan (and, for example, a corresponding surgeon agent). And such anupdated simulation may be used to more closely align with the liveprocedure than the original simulation, for example.

In an example, the surgical data system 31502 and/or the simulationdevice 31500 may be used capture simulated and/or real-world snapshotsof the procedure. Such snapshots may be recalled at a later point inprocedure, at a time for example, when one or more organs and/orretractions are distorting and/or occluding the surgeon's view. Forexample, the operative view may be augmented with one or more snapshotsfrom a preferred point in time, such as a preferred point in timeassociated with indocyanine green-based (ICG-based) anatomic delineationclarity. And once a critical structure is identified, the surgical datasystem 31502 and/or the simulation device 31500 may be used to re-apply(e.g., overlay) the snapshot as last viewed in the simulation.

In an example, the surgical data system 31502 and/or the simulationdevice 31500 may be used to archive certain alternative steps insimulation, such as alternate surgeon choices of approach, retraction,instrument usage, and the like. Such simulated alternative tasks may beaccessed during the procedure. A simulation may be updated to align withthe current state of the live procedure and then re-run with a selectedalternative to provide metrics related to expected results. Such re-runsimulations may be re-run automatically with this new information. Andthe surgical data system 31502 and/or the simulation device 31500 mayincorporate such new information to update procedure forecasting.

In an example, new and/or updated simulations may be constructed “on thefly” during a live procedure. Such simulations may enable a surgeon toinfer patient responses (e.g., micro-outcomes) and to explore theimplication of certain granular surgical step choices. For example, sucha simulation may enable a surgeon to determine a cause and/or correlatedresponse to an activity and/or observed behavior in the live procedure.

In an example, the surgical data system 31502 and/or the simulationdevice 31500 may be used compare to work backwards from a desiredoutcome and identify one or more steps and/or boundary conditions thatare associated with the desired outcome. For example, a comparison,using forward and backward simulation steps, may enable a surgeon toestimate the similarity between an actual and simulated surgicalapproach and its relation to a particular objective.

The operation of the simulation in concert with the live surgery may becoordinated by the surgical data system 31502, for example. Theoperation of the simulation in concert with the live surgery may becoordinated with reference to one or more procedure datasets.

Illustrating a use of the surgical data system 31502 and/or thesimulation device 31500, the surgical display 31510 may show the presentlive surgery. The surgical display 31502 may be displaying a currenttask associated with upper lobe manipulation in a thoracic lobotomy. Asshown, the surgeon's present technique is causing poor visualization ofthe pulmonary vein and/or pulmonary artery structures on the surgicaldisplay 31502. At this point, the surgeon may access a simulation of thecorresponding task via the simulation human user interface 31506. Thestored simulation may include a simulated based on the same and/orsimilar anatomy, performed by the same surgeon, performed by an expertsurgeon, simulated with the same and/or similar instruments, and thelike. The present task of the live procedure may be coordinated (e.g.,coordinated via common task-based indexing) with a corresponding portionof the stored simulation. In response to the request, the surgical datasystem 31502 and/or the simulation device 31500 may retrieve theappropriate portion of the simulation and present it to the surgeon forinteraction. As shown, the simulation may inform the surgeon of animproved instrument technique that may enhance visualization of thepulmonary vein and/or pulmonary artery structures by showing thesimulation of the technique via the simulation human user interface31506.

The operation of the simulation in concert with the live surgery may becoordinated with reference to a procedure plan for the simulatedprocedure, the procedure data associated with the actual simulation asexecuted, the procedure plan data for the live surgical procedure, thesituational awareness data from the live surgical procedure, and thelike.

FIG. 13 illustrates time-based surgical and simulation data. Here,example procedure plan data for the simulated procedure 31516, exampleprocedure data associated with the actual simulation as executed 31518,example procedure plan data for the live surgical procedure 31520, andexample situational awareness data from the live surgical procedure31522 are illustrated as a series of tasks with reference to a timeline31524. The datasets 31516, 31518, 31520, 31522 may be similarlystructured. For example, the datasets 31516, 31518, 31520, 31522 may bestructured according to a common and/or compatible data structure, suchas the example data structure disclosed herein with reference to FIG.11A.

The simulated procedure plan 31516 planned for task 1 to end and fortask 2 to begin at time T1. Similarly, when the simulated procedure wasexecuted as a simulation, for example before the actual live procedure,the simulated task 1 ended and simulated task 2 began at time T1, asrecorded by the simulation data 31518. In this example, the procedureplan data 31520 for the live procedure indicated that the task 1 wasplanned to end and task 2 was planned to begin at time T1. However,during the live surgical procedure, as indicated by the surgicalsituational awareness data 31522, task 1 ended and task 2 began at timeT2, which is different and/or after time T1.

Here, with a time-based reference, a surgeon retrieving a copy of thesimulated procedure for viewing and/or interacting during the liveprocedure would face mismatched timing. Such a surgeon may sync theplayback of the simulation to the live procedure by a jog operationforward and/or backward to find the appropriate time T1 in thesimulation that matched the corresponding task in the live surgery attime T2.

FIG. 14 illustrates a task-based indexing of the time-based surgical andsimulation data. Here, example procedure plan data for the simulatedprocedure 31526, example procedure data associated with the actualsimulation as executed 31528, example procedure plan data for the livesurgical procedure 31530, and example situational awareness data fromthe live surgical procedure 31532 are illustrated as a series of taskswith reference to a task-based index 31534. The datasets 31526, 31528,31530, 31532 may be similarly structured. For example, the datasets31526, 31528, 31530, 31532 may be structured according to a commonand/or compatible data structure, such as the example data structuredisclosed herein with reference to FIG. 11A. For example, the datasets31526, 31528, 31530, 31532 may be structured with a task identifierfield that provides unique identifier for individual tasks. For example,datasets 31526, 31528, 31530, 31532 may be structured with a task typefield that is a common key for data elements that contain informationabout various planned, simulated, or actual instances of the same task.

In the simulated procedure plan 31526, the transition for planned task 1to end and/or for planned task 2 to begin may be indexed at index 01.Similarly, when the simulated procedure was executed as a simulation,the end of simulated task 1 and/or the start of simulated task 2 may beindexed at index 01 in the simulation data 31528. Likewise, theprocedure plan data 31530 for the live procedure indexes the end of task1 as planned and the start of task 2 as planned with index 01. Andduring the live surgical procedure, as indicated by the surgicalsituational awareness data 31532, the end of task 1 and/or the beginningof task 2 may be indexed with index 01.

The indexing may be based on the transition from one task to asubsequent task, for example. The indexing may be based on completion ofa task. The indexing may be based on the start of a task. The indexingmay be common across different data sets. For example, a task may havecommon indexing for each corresponding instance of that task indifferent data sets. The procedure data structure may enable such commonindexing. In an example, a transition from a particular task to thesubsequent task may be commonly indexed with corresponding instances ofthe task and subsequent task in other data sets regardless of the timeplanned and/or actually elapsed associated with the task. Similarly, thetransition from a particular simulated task to the subsequent simulatedtask may be commonly indexed with corresponding instances of the livetask and subsequent live task in another data set regardless of the timecycle and/or framerate of the simulated task.

Here, with task-based indexing, a surgeon retrieving a copy of thesimulated procedure for viewing and/or interacting during the liveprocedure may use the common index, index 01 for example, to skip to theappropriate portion of the simulation, such as the portion of thesimulation that corresponds to the present point in the live procedure.

In an embodiment, the surgeon may run and/or retrieve one or morealternative simulated tasks. The alternative simulated task may beindicated by corresponding procedure plan data for the alternatesimulated task 31536. The alternate simulated task may be indicated bythe simulation data of the alternate simulated task 31538. Here, thestart of the corresponding alternate task as planned for simulation maybe similarly indexed at index 01. The start of the correspondingalternate task as simulated may be similarly indexed at index 01. Suchalternatives may be recorded via a common data structure. For example,such alternatives may be accommodated by the data structure disclosedherein with reference to FIG. 11B for example. For example, tasks thatmay serve as alternatives for one another, such as at branching and/ordecision points, may be commonly indexed.

Index information may be stored as part of the disclosed datasets 31526,31528, 31530, 31532, 31536, 31538, for example. Index information may bestored apart from the disclosed datasets 31526, 31528, 31530, 31532,31536, 31538, for example. Index information may be stored in acentralized data base, for example. Index information in a centralizeddatabase may provide cross referencing to unique task identifierspresent in the disclosed data sets 31526, 31528, 31530, 31532, 31536,31538, for example.

FIG. 15 is a flow diagram of example process for providing simulationsupport in a live surgical procedure. At 31540, a procedure plan for alive surgical procedure may be identified. For example, the procedureplan for the live surgical procedure may be identified, created, copied,modified, or the like, such as part of a pre-surgical planning process.Such a procedure plan for the live surgical procedure may be used by asurgical data system in managing and/or supporting the live surgicalprocedure. For example, such a procedure plan may be used by thesurgical data system to support situational awareness functionality.

At 35142, a stored simulation may be identified. The stored simulationmay be a simulation of a surgical procedure that corresponds to theprocedure plan identified at 31540, for example. The stored simulationmay be a simulation of a surgical plan that corresponds to the procedureplan by having common tasks, for example. The stored simulation may be asimulation of a surgical plan that corresponds to the procedure plan byhaving setup, such as patient anatomy, for example. The storedsimulation may be a simulation of a surgical plan that corresponds tothe procedure plan by having tasks that may serve as alternatives to thetasks set forth in the procedure plan, for example.

In an example, the procedure plan may include a set of tasks. The storedsimulation may include information indicative of simulated activity. Theinformation of simulated activity may be indexed according to the set oftasks in the procedure plan.

At 31544, a present portion of the live surgical procedure may bedetermined. For example, the present portion of the live surgicalprocedure may be determined from information received during the livesurgical procedure. For example, the present portion of the livesurgical procedure may be determined from information received duringthe live surgical procedure at a surgical data system, such as asurgical hub 106 for example. For example, the present portion of thelive surgical procedure may be determined from surgical situationalawareness information received during the live surgical procedure. Forexample, the present portion of the live surgical procedure may bedetermined from task index information received during the live surgicalprocedure.

At 31546, a portion of the stored simulation that corresponds to thepresent portion of the live surgical procedure may be retrieved. Forexample, the portion of the stored simulation that corresponds to thepresent portion of the live surgical procedure may be retrieved based onan analysis of the information received during the live surgicalprocedure, at 31544 for example. For example, the portion of the storedsimulation that corresponds to the present portion of the live surgicalprocedure may be retrieved based on a common index between the portionof the stored simulation and the present portion of the live surgicalprocedure.

In an example, the present portion of the live surgical procedure may bedetermined, at 31546, according to a present task (such as by an indexof the present task) from the procedure plan identified at 31542. Aportion of the store simulation (e.g., a portion of informationindicative of simulated activity) may be retrieved based on an index ofthe present task.

At 31546, the portion of the stored simulation that corresponds to thepresent portion of the live surgical procedure may be retrieved. Forexample, the portion of the stored simulation that corresponds to thepresent portion of the live surgical procedure may be retrieved from thestored simulation. For example, the portion of the stored simulationthat corresponds to the present portion of the live surgical proceduremay be retrieved from the stored simulation in a simulation device, suchas simulation device 31500, for example.

At 31546, the portion of the simulation that corresponds to the presentportion of the live surgical procedure may be presented for userinteraction. For example, the portion of the simulation that correspondsto the present portion of the live surgical procedure may be presentedfor user playback. For example, the portion of the simulation thatcorresponds to the present portion of the live surgical procedure may bepresented for a user to perform simulated activities. For example, theportion of the simulation that corresponds to the present portion of thelive surgical procedure may be presented for a user to consideralternative surgical tasks.

The portion of the simulation that corresponds to the present portion ofthe live surgical procedure may be presented for a user interaction bythe surgical data system, such as surgical data system 31502, forexample. The portion of the simulation that corresponds to the presentportion of the live surgical procedure may be presented for a userinteraction by the simulation device, such as simulation device 31500,for example. The portion of the simulation that corresponds to thepresent portion of the live surgical procedure may be presented for auser interaction by the simulation device in cooperation with a surgicaldata system, such as by simulation device 31500 in cooperation withsurgical data system 31502, for example.

For example, control input may be received during the live procedure.The control input may be for the portion of the simulation thatcorresponds to the present portion of the live surgical procedure. Basedon the control input, a live simulation may be executed. The livesimulation may be executed during the live procedure. For example, thelive simulation may correspond to the portion of the stored simulationthat corresponds to the present portion of the live surgical procedure.In an example, such a control may include a modification of the useractivity in the stored simulation. For example, the modification of theuser activity may include modifying any of selection, instrumentconfiguration, technique selection, application location, or the like.In an example, such a control may include a modification of thesimulation settings of the stored simulation. For example, themodification of the settings may include modifying the simulatedanatomy.

A visualization of the present portion of the live surgical proceduremay be presented. A presentation of the retrieved portion of simulationmay be presented. The visualization of the present portion of the livesurgical procedure may be presented concurrently with a presentation ofthe portion of the simulation that corresponds to the present portion ofthe live surgical procedure. In an example, a user control may include atimeline user control to view the stored simulation at a time other thanthat which corresponds to the present portion of the live surgicalprocedure. In an example, a different portion of the stored simulationmay be retrieved based on a user's selection of the timeline control.

In an example, a user control may include a task-based index usercontrol to view the stored simulation at a task other than that whichcorresponds to the present portion of the live surgical procedure. In anexample, a different portion of the stored simulation may be retrievedbased on a user's selection of the task-based index user control. In anexample, a user control may include a procedure plan user control. Theuser may use the procedure plan user control to view a different portionof stored simulation that corresponds to that selected by the procedureplan user control. For example, a different portion of the storedsimulation may be retrieved based on a user's selection of the procedureplan control.

The process flow at 31540-31548 may be performed at any suitableconfigured processor, such as a processor of the surgical data system31502, a processor of the simulation device 31500, a processor of thehuman interface device 31506, and the like.

Simulation support may refer to the use of a surgical simulation duringa live surgical procedure so as to provide active guidance to thesurgeon during a surgery. For example, simulation support may refer tohaving a simulation environment that runs simultaneously andcontinuously with the live surgical procedure. For example, simulationsupport may refer to a simulation which is not actively being runsimultaneously with the live surgical procedure, but one which can bequeried so as to present a portion of a simulation which has been run atleast once. Each of these embodiments can be considered to providesupport, or guidance, to a surgeon during a live surgical procedure.

User interaction may also refer to user guidance. In other words, theportion of the simulation may be presented or displayed to the surgeonfor the purposes of guiding the surgery. It may be understood that alive surgical procedure may refer to a surgical procedure that ishappening in real-time. In other words, the live surgical procedure canbe happening concurrently with the operation of the simulation.

The processor may be configured to determine a portion of the livesurgical procedure through information received (e.g. through imageprocessing or through information received from a surgical hub,configured to determine surgical context).

1. A device for providing simulation support in a live surgicalprocedure, the device comprising: a processor configured to: identify aprocedure plan for a live surgical procedure; identify a storedsimulation of a surgical procedure that corresponds to the procedureplan; determine, from information received during the live surgicalprocedure, a present portion of the live surgical procedure; retrieve,from the stored simulation, a portion of the stored simulation thatcorresponds to the present portion of the live surgical procedure; andpresent, for user interaction, the portion of the simulation thatcorresponds to the present portion of the live surgical procedure. 2.The device of claim 1, wherein the procedure plan comprises a set oftasks, and wherein the stored simulation comprises informationindicative of simulated activity indexed according to the set of tasks.3. The device of claim 2, wherein the processor is further configured todetermine the present portion of the live surgical procedure bydetermining a present task from the procedure plan.
 4. The device ofclaim 3, wherein the processor is further configured to retrieve aportion of the simulation that corresponds to the present portion of thelive surgical procedure by retrieving a selected portion of theinformation indicative of simulated activity that is indexed the presenttask.
 5. The device of claim 1, wherein the processor is furtherconfigured present a visualization of the present portion of the livesurgical procedure concurrently with a presentation of the portion ofthe simulation that corresponds to the present portion of the livesurgical procedure.
 6. The device of claim 1, wherein the user controlcomprises a timeline user control to view the stored simulation at atime other than that which corresponds to the present portion of thelive surgical procedure.
 7. The device of claim 6, wherein the processoris configured to retrieve a different portion of the stored simulationbased on a user's selection of the timeline control.
 8. The device ofclaim 1, wherein the user control comprises a procedure plan usercontrol to view a different portion of stored simulation thatcorresponds to that selected by the procedure plan user control.
 9. Thedevice of claim 8, wherein the processor is configured to retrieve adifferent portion of the stored simulation based on a user's selectionof the procedure plan control.
 10. The device of claim 1, wherein theprocessor is further configured to receive control input during the liveprocedure, and to execute, with the control input, a live simulationthat corresponds to the portion of the stored simulation thatcorresponds to the present portion of the live surgical procedure. 11.The device of claim 10, wherein the control input is a modification ofthe user activity in the stored simulation.
 12. The device of claim 11,wherein the user activity comprises any of instrument selection,instrument configuration, technique selection, or application location.13. The device of claim 10, wherein the control input is a modificationof the simulation settings.
 14. The device of claim 10, wherein thecontrol input is a modification of simulated anatomy.
 15. A method forproviding simulation support in a live surgical procedure, the methodcomprising: identifying a procedure plan for a live surgical procedure;identifying a stored simulation of a surgical procedure that correspondsto the procedure plan; determining, from information received during thelive surgical procedure, a present portion of the live surgicalprocedure; retrieving, from the stored simulation, a portion of thestored simulation that corresponds to the present portion of the livesurgical procedure; and presenting, for user interaction, the portion ofthe simulation that corresponds to the present portion of the livesurgical procedure.
 16. The method of claim 15, wherein the procedureplan comprises a set of tasks, and wherein the stored simulationcomprises information indicative of simulated activity indexed accordingto the set of tasks.
 17. The method of claim 16, further comprisingdetermining the present portion of the live surgical procedure bydetermining a present task from the procedure plan.
 18. The method ofclaim 17, further comprising retrieving a portion of the simulation thatcorresponds to the present portion of the live surgical procedure byretrieving a selected portion of the information indicative of simulatedactivity that is indexed the present task.
 19. A device for providingsimulation support in a live surgical procedure, the device comprising:a processor configured to: identify a procedure plan for a live surgicalprocedure; identify a stored simulation of a surgical procedure thatcorresponds to the procedure plan; retrieve, from the stored simulation,a portion of the stored simulation that corresponds to a portion of thelive surgical procedure; present, for user interaction, the portion ofthe stored simulation that corresponds to the present portion of thelive surgical procedure; receive control input during the liveprocedure; and execute, with the control input, a live simulation thatcorresponds to the portion of the stored simulation that corresponds tothe portion of the live surgical procedure.
 20. The device of claim 19,wherein the control input is any of a modification of user activity inthe stored simulation, a modification of the simulation settings, or amodification of simulated anatomy.